Are current reasoning models optimal for test-time scaling? 🌠 No! Models make the same incorrect guess over and over again. We show that you can fix this problem w/o any crazy tricks 💫 – just do weight ensembling (WiSE-FT) for big gains on math! 1/N

New unlearning work at #ICLR2025! We give guarantees for unlearning a simple class of language models (topic models), and we further show it's easier to unlearn pretraining data during fine-tuning, without even modifying the base model. Paper: arxiv.org/abs/2411.12600 🧵:

What is the role of memory for modeling time dependent PDEs? I will be at ICLR presenting our paper (Oral) where we study when it is beneficial for modeling time-dependent PDEs! 🔗openreview.net/forum?id=o9kqa… [Oral]: Thu 24 Apr 10:30 am @ Session 1E [Poster]: Thu 24 Apr 3 pm #617

Thrilled that Abhishek Panigrahi is presenting our paper (joint with Bingbin Liu, Sadhika Malladi, Andrej Risteski) on the benefits of progressive distillation as an oral at #ICLR2025. Talk details below ⬇️ Also check out our blog post: unprovenalgos.github.io/progressive-di…

Excited to present our recent findings on "catastrophic overtraining" where more pre-training data (shockingly) can lead to worse downstream models.

Our paper on how overtraining LLMs can make fine-tuning harder won awards at two different #ICLR2025 workshops! I'm honored and thrilled! Outstanding paper @ SCOPE Entropic Paper Award @ ICBINB

Our work on the surprising negative effects of pre-training LMs for longer has received awards from both of these workshops and has been accepted to ICML 25! One of my first papers as a senior author :) arxiv.org/abs/2503.19206

Announcing the 1st Workshop on Methods and Opportunities at Small Scale (MOSS) at ICML Conference 2025! 🔗Website: sites.google.com/view/moss2025 📝 We welcome submissions! 📅 Paper & jupyter notebook deadline: May 22, 2025 Topics: – Inductive biases & generalization – Training

Honored to be an invited speaker at this COLT workshop :) do consider submitting relevant work, and see you in Lyon!

I was really grateful to have the chance to speak at Cohere Labs and ML Collective last week. My goal was to make the most helpful talk that I could have seen as a first-year grad student interested in neural network optimization. Sharing some info about the talk here... (1/6)

The abstract submission deadline for FoPt has been extended to the 21st of May (11:59pm UTC). Submission website: openreview.net/group?id=learn…

Data selection and curriculum learning can be formally viewed as a compression protocol via prequential coding. New blog (with Allan Zhou ) about this neat idea that motivated ADO but didn’t make it into the paper. yidingjiang.github.io/blog/post/curr…

Adam is similar to many algorithms, but cannot be effectively replaced by any simpler variant in LMs. The community is starting to get the recipe right, but what is the secret sauce? Robert M. Gower 🇺🇦 and I found that it has to do with the beta parameters and variational inference.