Super excited to have this out! Was great to work on this with Yusuf Kocyigit supervised by Dieuwke Hupkes and figure out the best post-hoc methods for identifying eval contamination + measure its effects on performance. A short 🧵

Why do we treat train and test times so differently? Why is one “training” and the other “in-context learning”? Just take a few gradients during test-time — a simple way to increase test time compute — and get a SoTA in ARC public validation set 61%=avg. human score! ARC Prize

I am looking for Machine Learning Intern for the Spring or Summer terms at the AI Institute for scaling our robot policy learning stack. Apply here and DM me! jobs.lever.co/bostondynamics…

🌐 Meet MetricX-24, our SOTA machine translation evaluation metric and a successor to the successful MetricX-23. 🚀 Now open-source in PyTorch/Transformers! 🎉 Ready to take this top performer in the WMT24 Metrics Shared Task for a spin? 🔗 Code: github.com/google-researc…

Ekin Akyürek (Ekin Akyürek) builds tools for understanding & controlling algorithms that underlie reasoning in language models. You’ve likely seen his work on in-context learning; I'm just as excited about past work on linguistic generalization & future work on test-time scaling.

Our work got accepted to ICML! Looking forward to sharing more about this project with everyone this summer!

Pulling this opportunity on research agent evaluation up one more time! The official title of the position will be "Senior research technician". Feel free to email either Sebastian Schuster or me directly if you have any questions. Link for more detailed info and where to apply in 🧵

🗺️ Are we making our #LLMs multilingual, or anglocentric? Much work brings languages closer to English, but that comes at the cost of crucial #cultural nuance. HyoJung Han tackles this trade-off with surgical steering, adapting LLMs to cultural contexts at inference time.

This is Gemini 3: our most intelligent model that helps you learn, build and plan anything. It comes with state-of-the-art reasoning capabilities, world-leading multimodal understanding, and enables new agentic coding experiences. 🧵

I’m really excited about our release of Gemini 3 today, the result of hard work by many, many people in the Gemini team and all across Google! 🎊 We’ve built many exciting new product experiences with it, as you’ll see today and in the coming weeks and months. You can find it

My lab at BU is recruiting PhD students and possibly a postdoc this year! We study humans & machines, centered around topics like meaning, generalization, evaluation methods and design, and the nature of computation and representation that underlie language and cognition. 🫴🫴