Alex Wettig (@_awettig) 's Twitter Profile
Alex Wettig

@_awettig

PhD in AI @princeton

ID: 1549104683955785728

linkhttps://www.cs.princeton.edu/~awettig/ calendar_today18-07-2022 18:52:37

96 Tweet

637 Followers

332 Following

Alex Wettig (@_awettig) 's Twitter Profile Photo
Alex Wettig
@_awettig

2 months ago

Simple strategy: (1) keep pre-training with HQ mix of long & short documents (2) quick instruction-tuning with ONLY short UltraChat We find: avg. performance on our long-context evals keeps improving with increasing continual pre-training budgets

Simple strategy: (1) keep pre-training with HQ mix of long & short documents (2) quick instruction-tuning with ONLY short UltraChat We find: avg. performance on our long-context evals keeps improving with increasing continual pre-training budgets