I will touch on some work we've (Christopher Wang, Sabera Talukder, Vighnesh Subramaniam, Saraswati, Yisong Yue, Boris Katz, Andrei Barbu) done on pretraining transformers across variable input layouts for 🔥 decoding and interpretability. Read more here: arxiv.org/abs/2406.03044

Late last year TOTEM was: 1⃣ Accepted to TMLR 🥳 openreview.net/pdf?id=QlTLkH6… 2⃣ Crossed 100 ⭐️s on Github github.com/SaberaTalukder… 3⃣ Released a video summary (🙏 Steven Brunton) youtube.com/watch?v=OqrCpd… Glad you're feeling the tokens! More coming soon 😉

Proud of Geeling Chau leading this new work on neural time series modeling, which just received Oral (top 1.8%) ICLR 2026 2025! Always a delight to support her persistence! Arxiv version: arxiv.org/pdf/2406.03044

1/5 LLMs are notoriously not great at 3D perception … but what if there were an alternative path to scaling 3D VQA? 👀 Introducing VADAR: a new program synthesis approach for 3D spatial reasoning! glab-caltech.github.io/vadar/

Introducing Kyvo! 🚀 – a decoder-only LLM that aligns text, images & structured 3D scenes token-by-token. From a single image, it reconstructs individual 3D shapes and their locations, renders & edits scenes, answers spatial questions, and more. 💻: glab-caltech.github.io/kyvo/

Honored to receive the mentoring award from the Grad Student Advisory Board at Caltech EAS. Thanks to the students who nominated me, especially Sabera Talukder!

Incredibly well deserved, Yisong Yue is simply the best! Also huge shout out to Hongkai Zheng & Raul Astudillo who also wrote wonderful letters and the rest of the Yue Lab Caltech + friends who endorsed 🦫🧡

Inspired time Laude Institute, huge congrats to Andy Konwinski on the launch! Humbled to be asked to speak on the dos of impact research in front of such 🤩 researchers!! P.S. 👀 the where's waldo moment of my favorite +1 (Yisong Yue) being (I hope 😅) a proud academic dad 🪞📸

Choose your fighter.

Collaboration is important for impactful work, but it is not enough! Impact comes from the collaboration between two types of people. The generators and discriminators (pun very much intended 😉). The generators dream big, swing for the fences, and don’t listen when they are

Super excited about 🥋judo🥋, our new open-source sampling-based MPC toolbox! 🖥️Try in under 30s: pip install judo-rai judo <click link> 🧵thread below👇

I know of folks training on non-specialized hardware and easily clearing 1,000,000 tokens/sec. So while impressive, I'm not convinced that specialized hardware is the way to go if the selling point is simply faster on the tokens/sec measure (they claim 500,000 btw). What is

many such cases

Settle a recent debate for me! Are constraints (compute, data scale, small teams, etc.) in academia sparking creativity or leading to an increasingly insurmountable knowledge gap?

We really enjoy developing Graphite (we've moved to @graphite), particularly their stacking functionality! It makes a huge difference with larger teams. Bonus points for their support of academics 🫶🫶🫶

Crazy that I started grad school with one 2080Ti, and am ending with 100s of V4s

it Yisong Yue

Name something more mind-numbing yet integral than data wrangling (cleaning, augmenting, labeling, etc. )... I'll wait.

You cannot tell someone both what to do and how to do it. People need creativity in at least one layer of this stack to feel motivated.

I love ubiquitous principles that transcend domains. For instance, consistency trumps any other type of forward progress. This is true in athletics, research, entrepreneurship, etc. What are the other ubiquitous principles you've observed (good or bad!)?