I’m joining the Columbia Computer Science faculty as an assistant professor in fall 2025, and hiring my first students this upcoming cycle!! There’s so much to understand and improve in neural systems that learn from language — come tackle this with me!

We already have a fun retro promotional video for this paper: youtu.be/O7oD_oX-Gio?fe…

We talk a lot about the potential of AI for applications, like AI for Education. But actual progress requires that we hill-climb on realistic, hard tasks. Are there any? 🔽 Bridge, Backtracing, and Teacher Coach are 3 real-world AI for Education datasets that are far from

I'm joining Cornell University this fall as an Assistant Professor of Computer Science! Looking forward to work with students and colleagues Cornell Computer Science, @cornellCIS on generative models, controllable generation, and creative applications like #musictechnology

🔥 Super excited for isaac 🧩 to join us at Normal Computing 🧠🌡️ to work on crazy #dspy stuff for semiconductors! 🔥

An LLM memorization riddle: A Pythia-6.9B checkpoint generates the following Output, which occurs only 1 time in the Pile. Is this a verbatim memorization?

Do you like LLMs? Do you also like for loops? Then you’ll love our new paper! We scale inference compute through repeated sampling: we let models make hundreds or thousands of attempts when solving a problem, rather than just one. By simply sampling more, we can boost LLM

Come see #ACL2024‘s most beautiful poster, being presented by Julie Kallini ✨ right now at poster 7! Refuting Chomsky‘s NYT Op-Ed with experimental work as an added bonus ✨🌈

Aligning Language Models with preferences leads to stronger and safer models (GPT3 → ChatGPT). However, preferences (RLHF) contain irrelevant signals, and alignment objectives (e.g. DPO) can actually hurt model performance. We tackle both, leading to a ~2x performance boost.

Congratulations to Julie Kallini ✨ and coauthors for an #ACL2024 Best Paper Award for their paper Mission: Impossible Language Models!

I'm incredibly proud that Aya received #ACL2024 Best Paper Award 🥹. Huge congratulations to the Aya team and Cohere For AI community who make this possible by for extending frontiers of LLMs to multilingual, building Aya Model and Aya Dataset 🌿🌏

🤩So honored to receive TWO paper awards ACL 2025, huge shoutout to my amazing collaborators🤩!!! 🏆Best Social Impact Paper🏆: persuasive jailbreaker arxiv.org/abs/2401.06373 🏆Oustanding Paper🏆: persuasive misinformation arxiv.org/abs/2312.09085 #ACL2024

causalgym won an area chair award and outstanding paper award at ACL 😁 thanks to my very cool advisors Christopher Potts and Dan Jurafsky

.Stanford NLP Group awards at #ACL2024 ▸ Best paper award Julie Kallini ✨ et al ▸ Outstanding paper award Aryaman Arora et al ▸ Outstanding paper award Weiyan Shi et al ▸ Best societal impact award Weiyan Shi et al ▸ 10 year test of time award Christopher Manning et al Congratulations! 🥂

Honored to receive Best Paper Award at #ACL2024 for “Mission: Impossible Language Models”! 🚀 Big thank you to my co-authors Isabel Papadimitriou Richard Futrell Kyle Mahowald Christopher Potts!

Some personal news: I'm thrilled to have joined @Databricks Databricks Mosaic Research as a Research Scientist last month, before I start as MIT faculty in July 2025! Expect increased investment into the open-source DSPy community, new research, & strong emphasis on production concerns 🧵.

The Linear Representation Hypothesis is now widely adopted despite its highly restrictive nature. Here, Csordás Róbert, Atticus Geiger, Christopher Manning & I present a counterexample to the LRH and argue for more expressive theories of interpretability: arxiv.org/abs/2408.10920

A short story of fast progress: NVIDIA released an ≈8B parameter model they called Megatron in 2019, and five years later they have released an ≈8B model they call Minitron. (I did round off an entire BERT-large for the 2019 model.)

Do you want to select great LLM pretraining data but don’t have 1000 H100s for a ton of mixture experiments? What about a method that requires none of your own training, matches the best known existing method, and has some nice theory? New preprint: Perplexity Correlations

If I finetune my LM just on responses, without conditioning on instructions, what happens when I test it with an instruction? Or if I finetune my LM just to generate poems from poem titles? Either way, the LM will roughly follow new instructions! Paper: arxiv.org/pdf/2409.14254