L1 regularization for sparse solutions - as usually taught - is actually terrible in practice! I’m always surprised how few people know this. To get good results, retrain with the sparsity pattern found from the initial L1 run, but without the regularizer. Works much better.

My Gemma-3 analysis: 1. 1B text only, 4, 12, 27B Vision + text. 14T tokens 2. 128K context length further trained from 32K 3. Removed attn softcapping. Replaced with QK norm 4. 5 sliding + 1 global attn 5. 1024 sliding window attention 6. RL - BOND, WARM, WARP Detailed analysis:

Nathan Lambert I know you all uploaded GGUFs, but also just uploaded other GGUF formats + dynamic 4bit bitsandbytes and general 4bit BnB versions! Dynamic 4bit BnB: huggingface.co/unsloth/OLMo-2… 4bit BnB: huggingface.co/unsloth/OLMo-2… GGUFs: huggingface.co/unsloth/OLMo-2… Fantastic true open source model!

This paper also recommended for understanding GRPO. TLDR is that the output-length normalization in DeepSeek GRPO is making models not penalize repetitive behaviors while rewarding shorter correct responses. Same intuition as the last RL paper I posted. Writeup soon.

Today, we're announcing our $50M Series A and sharing a preview of Ember - a universal neural programming platform that gives direct, programmable access to any AI model's internal thoughts.

And to finish off, Lectures 21 - 23: - AI for Mathematics: youtu.be/ToY57HgQKXA - Multimodal I (CLIP / Llava): youtu.be/5uI5WOpq8LQ - Multimodal II (VQVAE / Chameleon): youtu.be/VismiXpCs_Y

first time someone talking about 2045 AGI timeline

We're missing (at least one) major paradigm for LLM learning. Not sure what to call it, possibly it has a name - system prompt learning? Pretraining is for knowledge. Finetuning (SL/RL) is for habitual behavior. Both of these involve a change in parameters but a lot of human

Our interpretability team recently released research that traced the thoughts of a large language model. Now we’re open-sourcing the method. Researchers can generate “attribution graphs” like those in our study, and explore them interactively.

Wrapped up Stanford CS336 (Language Models from Scratch), taught with an amazing team Tatsunori Hashimoto Marcel Rød Neil Band Rohith Kuditipudi. Researchers are becoming detached from the technical details of how LMs work. In CS336, we try to fix that by having students build everything:

It was a dream come true to teach the course I wish existed at the start of my PhD. We built up the algorithmic foundations of modern-day RL, imitation learning, and RLHF, going deeper than the usual "grab bag of tricks". All 25 lectures + 150 pages of notes are now public! 🧵

LLMs can be programmed by backprop 🔎 In our new preprint, we show they can act as fuzzy program interpreters and databases. After being ‘programmed’ with next-token prediction, they can retrieve, evaluate, and even *compose* programs at test time, without seeing I/O examples.

Asking a non-expert in AI but as a (discrete) mathematician: does anyone really know how deep learning works mathematically?

This is the syllabus of the course Geoffrey Hinton and I taught in 1998 at the Gatsby Unit (just after it was founded). Notice anything?

Munchkin you beauty

I wrote this last year, and it doesn't hurt to share it again. A phone-formatted ELBO TL;DR.

donk 3k to close the map

KFC - Kolmogorov Fan Club