What precision should we use to train large AI models effectively? Our latest research probes the subtle nature of training instabilities under low precision formats like MXFP8 and ways to mitigate them. Thread 🧵👇

Excited to share this work on understanding low-precision instabilities in model training! See our thread below for more details. Paper: arxiv.org/abs/2506.20752 Blogpost: tinyurl.com/lowprecinstabi…

Excited to be giving this talk at COLT tomorrow :) reach out if you want to chat about deriving useful theoretical insights into modern-day language models!

LoRA is amazing for finetuning large models cheaply, but WHERE you place the adapters makes a huge difference. Most people are just guessing where to put them (Attention, MLP, etc). Meet "PLoP" (Precise LoRA Placement) 🎯, our new method for automatic LoRA placement 🧵

[1/n] Discussions about LM reasoning and post-training have gained momentum. We identify several missing pieces: ✏️Post-training based on off-the-shelf base models without transparent pre-training data components and scale. ✏️Intermediate checkpoints with incomplete learning

Excited to share that I’ll be presenting two oral papers in this ICML—see u guys in Vancouver!!🇨🇦 1️⃣ arxiv.org/abs/2502.06768 Understanding Masked Diffusion Models theoretically/scientifically 2️⃣ arxiv.org/abs/2502.09376 Theoretical analysis on LoRA training

🧵1/7 Why do LLMs generalize so well, yet also hallucinate? 🤔 Our new paper argues they are two sides of the same coin, driven by a single powerful mechanism: Out-of-Context Reasoning (OCR). 🔗 arXiv: arxiv.org/abs/2506.10887

Great to see this one finally out in PNAS! Asymptotic theory of in-context learning by linear attention pnas.org/doi/10.1073/pn… Many thanks to my amazing co-authors Yue Lu, Mary Letey, Jacob Zavatone-Veth and Anindita Maiti

Excited to announce our workshop on Causality in Science at #NeurIPS2025! See you in San Diego 🌴🇺🇸

Pass by if you want to know about scaling up your model under distribution shifts of the training data. Take away: muP needs to be tuned to the optimal amount of feature learning that optimizes the forgetting/plasticity trade off.

First #ICML2025 conference proceeding (icml.cc/virtual/2025/p…)! We (Blake Bordelon, Jacob Zavatone-Veth, Cengiz Pehlevan) developed a simple model to better understand state representation learning dynamics in both artificial and biological intelligent systems! Comments appreciated!

A team from #KempnerInstitute, Harvard SEAS & Computer Science at UT Austin has won a best paper award at #ICML2025 for work unlocking the potential of masked diffusion models. Congrats to Jaeyeon (Jay) Kim @ICML, Kulin Shah, Vasilis Kontonis, Sham Kakade and Sitan Chen. kempnerinstitute.harvard.edu/news/kempner-i… #AI

ICML this week! Come by T PM Clarissa Lauditi's work on muP BNNs arxiv.org/abs/2502.07998 W AM, model of place field adaptation@mgkumar138, Jacob ZV biorxiv.org/content/10.110… W PM a model of LR transfer in linear NNs arxiv.org/abs/2502.02531 all from senior author Cengiz Pehlevan!

As Micah Goldblum and coauthors, we also found that small batches make SGD effective in LM training. It's cool that our papers came out around the same time, and each has a different perspective! Below, our take on why this happens. Our awesome team: Teodora Jonas Geiping

Come to HilD tomorrow International Conference on Minority Languages ! We have 4 posters on optimization: - In Search of Adam’s Secret Sauce - Is your batch size the problem? Revisiting the Adam-SGD gap in language modeling - On the Interaction of Noise, Compression Role, and Adaptivity under (L0,L1)-Smoothness

New in the #DeeperLearningBlog: the #KempnerInstitute's Mary Letey presents work recently published in PNAS that offers generalizable insights into in-context learning (ICL) in an analytically-solvable model architecture. bit.ly/4lPK15p #AI PNASNews (1/2)

blake taught me a bunch of stuff in my 1st year at Harvard, and he's an all around swell fella, you should work with him!

Very excited to be a part of this amazing collaboration! Thanks Simons Foundation for the support!

🚨New video! I talked about my research — 1️⃣ My work at Seoul National University 2️⃣ Our Outstanding Paper Award at ICML 2025 Also check out another podcast with Yilun Du — an amazing researcher & AP at Harvard CS! Harvard CS is growing fast 🫡

This winter I'm speaking at the LLM & Control workshop of CDC in Rio! If you are interested in this awesome combo, pls register!! sites.google.com/view/cdc2025-l…