1/Pretraining is hitting a data wall; scaling raw web data alone leads to diminishing returns. Today DatologyAI shares BeyondWeb, our synthetic data approach & all the learnings from scaling it to trillions of tokens🧑🏼‍🍳 - 3B LLMs beat 8B models🚀 - Pareto frontier for performance

🌻 Excited to announce that I’ve moved to NYC to start as an Assistant Prof/Faculty Fellow at New York University! If you’re in the area, reach out & let’s chat! Would love coffee & tea recs as well 🍵

この度、Forbes JAPANが選ぶ「世界を変える30歳未満の30人」に選出いただきました！ これからも精進して研究します。日本を再びロボットの中心地に！！！！！ 30 UNDER 30【ForbesJAPAN】 #u30fj

A bit late, but finally got around to posting the recorded and edited lecture videos for the **How to AI (Almost) Anything** course I taught at MIT in spring 2025. Youtube playlist: youtube.com/watch?v=0MYt0u… Course website and materials: mit-mi.github.io/how2ai-course/… Today's AI can be

Training a Whole-Body Control Foundation Model -- new work from my team at Agility Robotics A neural network for controlling our humanoid robots which is robust to disturbances, can handle heavy objects, and is a powerful platform for learning new whole-body skills learn more

Yesterday was my last day at xAI. It’s been an incredible ride for the past year and half, probably the most adventurous and fast growing period of my life so far. Best wishes for the team going forward. Looking forward to what’s next!

Coming from a computer vision background and now in sequence modeling, I’m often struck by how disconnected LLMs and vision feel. Our work, AUSM, treats video as language -- and it reveals a few blind spots we’ve overlooked.

I'm glad this collaboration happened and that the investigator agent could play a part in it!

1/Excited to share the first in a series of my research updates on LLM pretraining🚀. Our new work shows *distilled pretraining*—increasingly used to train deployable models—has trade-offs: ✅ Boosts test-time scaling ⚠️ Weakens in-context learning ✨ Needs tailored data curation

Deeply honored to be a part of MIT Tech Review Innovators Under 35 List this year. This recognition highlights our work on building robot intelligence that generalizes to unseen and unstructured human environments, executed with my friends & colleagues at NYU Courant & beyond.

Mind-blowing view of #AuroraBorealis on my flight from JFK—>SFO. The wildest part? I edited this shot on my phone at 35000 ft using freakn Nano Banana 🍌 via Google Gemini App ♊️ and posting this from the same altitude right now thanks to Delta WiFi. The future is officially here ✨

🚨Excited to introduce a major development in building safer language models: Safety Pretraining! Instead of post-hoc alignment, we take a step back and embed safety directly into pretraining. 🧵(1/n)

We can’t keep slapping alignment bandage 🩹 on harmful LLMs and calling it safety. Let’s fix the leak at the source by making LLMs safe by design. Introducing, Safety Pretraining🛡️

One thing years of memorization research has made clear: unlearning is fundamentally hard. Neurons are polysemantic & concepts are massively distributed. There’s no clean 'delete'. We need architectures that are "unlearnable by design". Introducing, Memorization Sinks 🛁⬇️

We just dropped Sonnet 4.5, the best coding model! Agents are truly here now -- autonomous task solving, complex multi-step tasks, parallel agents, combined with new tools and features and a lot more.. Check it out here 👇

💡Can we trust synthetic data for statistical inference? We show that synthetic data (e.g. LLM simulations) can significantly improve the performance of inference tasks. The key intuition lies in the interactions between the moments of synthetic data and those of real data

Very interesting insights into understanding when and why synthetic data (although imperfect and biased) can boost the performance of statistical inference!! 📈📈

Check out this great work from Emily!

Thank you Rohan Paul for highlighting our work!💫 Front-Loading Reasoning shows that inclusion of reasoning data in pretraining is beneficial, does not lead to overfitting after SFT, & has latent effect unlocked by SFT! Paper: arxiv.org/abs/2510.03264 Blog:

Amazing theoretical work on how to generate text-based synthetic data that will *actually* improve performance on statistical inference!! 🤩