We benchmarked 10 optimizers and found that the recent new optimizers still have limited speed up (~10%) over Adam at a "larger" scale (1.2B, 8x data than Chinchilla optimal). I guess that means more research to be done in this area!

New open-weight small embedding model surpass mGTE !! With Matryoshka Embedding 🪆

Impressive models! 🚀 Congrats!!! We now have new SOTA encoder backbone. Our mGTE-MLM-base outperforms XLM-R-base in 2024 summer (or jan by training), its a "semi-modern" encoder 😃. Glad to see it stands as one of only two baselines, alongside XLM-R.

Really excited to share that FreshStack has been accepted at #neurips25 D&B Track (poster)! 🥁🥁 Huge congratulations to all my Databricks Mosaic Research co-authors! Time to see you in San Diego! 🍻

-2016 (classic era): focus on data efficiency 2017-2025 (pretraining era): focus on compute efficiency 2026-: focus on data efficiency (again) The standard Transformer paradigm is optimized for compute efficiency. As we look at data efficiency, we'll see very different design

Should you fine-tune your embedding model? (Spoiler: probably not 𝘺𝘦𝘵) 𝘉𝘦𝘧𝘰𝘳𝘦 jumping into fine-tuning, ask yourself: is your retrieval pipeline actually failing because of domain-specific knowledge gaps, or could it be something simpler? Here's what to check first:

Two headache things I had with the existing Search API: 1) Needing an extra crawl API to get full context. 2) not very accurate filtering over date, which leads to search time contamination for time-sensitive data. Perplexity Search API looks like a good replacement.

dr. jack morris Search will always be necessary. If you wanted to teach a human to be as smart as possible, you'd teach them how to find information and critical thinking, instead of trying to teach them everything about everything.

LoRA makes fine-tuning more accessible, but it's unclear how it compares to full fine-tuning. We find that the performance often matches closely---more often than you might expect. In our latest Connectionism post, we share our experimental results and recommendations for LoRA.

We're announcing a new update to MTEB: RTEB It's a new multilingual text embedding retrieval benchmark with private (!) datasets, to ensure that we measure true generalization and avoid (accidental) overfitting. Details in our blogpost below 🧵

Context engineering is key to building LLM agents. Can we let agents actively manage their own context? We introduce Context-Folding, giving agents the ability to branch and compress their context. Trained with RL on Search and SWE task, it beats ReAct using 10× less context.

One More (Small) Thing: Introducing mxbai-colbert-edge-v0 17M and 32M. They are are the result of an easily reproducible way to train ColBERT models from scratch. They're strong, too: the 17M variant would rank first on the LongEmbed leaderboard for models under 1B parameters.

The main breakthrough of GPT-5 was to route your messages between a couple of different models to give you the best, cheapest & fastest answer possible. This is cool but imagine if you could do this not only for a couple of models but hundreds of them, big and small, fast and

At #PyBay25, Guido van Rossum demo'd a Python package for "structured RAG". During ingestion, it uses LLM to extract structured data (entities/topics/verbs) and stores in standard DB, and then retrieves by structuring the user query as well. Try it out at: github.com/microsoft/type…

Qwen Deep Research just got a major upgrade. ⚡️ It now creates not only the report, but also a live webpage 🌐 and a podcast 🎙️ - Powered by Qwen3-Coder, Qwen-Image, and Qwen3-TTS. Your insights, now visual and audible. ✨ 👉 chat.qwen.ai/?inputFeature=…

Hi all, I am hosting a dinner party on 11.5 at EMNLP this year! We've invited a bunch VCs and startup people, also fantastic panelist to talk about embodied AI and LLM agents. All are welcome to attend!!