XLM-R has been SOTA for 6 years for multilingual encoders. That's an eternity in AI 🤯 Time for an upgrade. Introducing mmBERT: 2-4x faster than previous models ⚡ while even beating o3 and Gemini 2.5 Pro 🔥 + open models & training data - try it now! How did we do it? 🧵

Apologies that I haven't written anything since joining Thinking Machines but I hope this blog post on a topic very near and dear to my heart (reproducible floating point numerics in LLM inference) will make up for it!

Excited to announce that Sophont has raised $9.22M in combined pre-seed+seed rounds! 🚀🔥 Led by Kindred Ventures, with participation from @delphi_ventures Upfront Ventures AICONIC VENTURES also @jeffdean, @logankilpatrick, clem 🤗 (via Factorial Capital), Lukas Biewald & others

At MedARC we are building a comprehensive suite of medical LLM evals, and we already have tons of volunteers and lots of great progress! The project started less than a week ago! Are there other medical LLM evals we should include?

🛒 RexBERT: ModernBERT except for E-commerce was just released by RAHUL BAJAJ et al! 4 base encoders (17M, 68M, 150M, 400M) trained on 2.3T tokens (with 350B E-commerce related tokens), easily outperforming base models on E-commerce tasks! Details in 🧵

Can one wire this kernel into an existing PyTorch model? Or is this locked to the Modular platform?

ModernBERT is competitive with EmbeddingGemma.

Many AI policy decisions are complicated. "Don't ban self-driving cars" is really not. Good new piece from Kelsey Piper, with a lede that pulls no punches:

It's great to see our efficient architecture improvements powering a new ecosystem of encoder models.

Vibe engineering also seems to better describe "LLM pair programming" then vibe coding.

This paper is one of the most astonishing feats of sustained data wizardry I have ever seen. Using data from Uber, they are able to estimate the roughness of every road in America and precisely estimate the value people place on it, and so much more. 1/

Nvidia missed the mark by not equipping the Spark with ~400GB/s memory bandwidth.

MacStudio you ask? Apple Engineering's **actual** time spent on PyTorch support has't given me confidence that PyTorch Mac experience would get anywhere close to NVIDIA's any time soon, if ever. The Meta engineers continue to do a huge amount of heavy-lifting for improving the

Excited to share our first paper: Scaling Vision Transformers for Functional MRI with Flat Maps We introduce a new approach to training fMRI neuroimaging foundation models and demonstrate a strict dataset power scaling law!

We've released our first Sophont paper! 🔥 We're bullish on the potential foundation models to help unlock novel clinical applications for brain & mental health. So we working on training better neuroimaging foundation models. We develop a novel approach for training fMRI

I think it's awesome that we have an author contributing the benchmark they developed to our MedARC LLM eval suite!

META RESEARCHERS WHO WERE LAID OFF: Hit me up if you wanna work on open-source LLMs and multimodal models for medicine and healthcare! This includes working on reasoning/RLVR/etc. and self-supervised training. It's very exciting research and very impactful too, come join!

Belated life update - I'm starting a company. No boundaries between low/high level systems engineering, research, design or product. No boundaries between AI compute and situated human collaboration and learning. More soon - DMs open if you want to connect.