It's my favorite kind of work: linear algebra insight + fast kernels. When playing w Muon a while ago, we were thinking why not speed it up by operating on the small square matrix X X^T instead of the large rectangular matrix X. Jack, Noah, and Berlin spent many months

Announcing Gram Newton-Schulz, a new way to implement Muon that's 2x faster Trick 1: rejigger Newton-Schulz to replace rectangular matmuls with square symmetric ones Trick 2: collaborate with CUDA kings Jack Zhang and Berlin Chen from Tri Dao's lab Blog: dao-ailab.github.io/blog/2026/gram…

oh, it has numerical issues, but they are fixed by restart x.com/jcz42/status/2…

now that's what i call a kernel trick

Special thanks to noahamsel for being an amazing theory mentor and collaborator - for context, we had independently derived different fast algorithms for Newton-Schulz, but his version ended up being strictly faster than our original algorithm (and theoretically better

Jack Zhang btw it seems that fp16 instead of bf16 is used, which one is better?

You Jiacheng Theoretically (and on some test cases), fp16 is better since it’s more precise, and the tradeoff of a smaller range doesn’t affect newton schulz since the magnitudes shouldn’t get so high that they exit fp16’s smaller range. In practice, we haven’t seen a training run where fp16

Fixing Berlin Chen tag, go follow him and checkout his other work on Mamba3 :)

Amazing! Was trying to get the fast symmetric kernels working in FlashMUON, only to be disappointed they only supported the symmetric matrices, apparently converting them small square matrices is still faster! github.com/nil0x9/flash-m…

Fast muon optimizer coming to consumer cards. All the code was written as matmul + epilogue so once the mainloop was implemented for Blackwell consumer cards, all the fancy symmetric matmul just works and get speed-of-light

Stereo depth is important in robotics, and relies heavily on synthetic data. But what actually makes for good synthetic data? In WMGStereo, we study dataset design and discover a powerful data recipe - just 500 samples of our data can match 40k Sceneflow samples! 🧵[1/7]

🚀SonicMoE🚀now runs at peak throughput on NVIDIA Blackwell GPUs 😃 54% & 35% higher fwd/bwd TFLOPS than the DeepGEMM baseline and 21% higher fwd TFLOPS than the triton official example. SonicMoE still maintains its minimum activation memory footprint: the same as a dense model

Congrats noahamsel!