OpenReview's LaTeX parser seems to be quite bad and it makes it very painful to be a reviewer sometimes. For example: "Assume $\hat{f}_n$ is smooth over $​ C_{n}$" can be parsed only if it's split into two paragraphs, which makes no sense. Can you please fix this? Open Review

Shuvomoy Das Gupta (Columbia) and I (UCLA) are starting an optimization seminar series! Our first speaker, Aaron Defazio (Meta), presented Schedules & Schedule-Free Learning. Aaron will give his NeurIPS 2024 Oral work next week. This is a longer version. (Video link in reply)

My book is (at last) out, just in time for Christmas! A blog post to celebrate and present it: francisbach.com/my-book-is-out/

I'm happy / thrilled / delighted / excited / passionate to share that our paper "A Stochastic Approach to the Subset Selection Problem via Mirror Descent" will appear in ICLR 2025 this year. Selecting a good subset of objects is fundamental not only in CS, but also in ML, i.e.

What students expect from ML job: analysis of sharpness effecting generalization bound analysis of NTK parameterization on convergence sub quadratic attention novel convolution architecture novel SDE solver for fast sampling rao-blackwellized gradient estimator for faster

1/n In new work, we draw connections between accelerated SGD and various recent optimizers including AdEMAMix, Schedule-Free optimizers and MARS, and use it to design ‘Simplified-AdEMAMix’ which matches performance of AdEMAMix without any extra momentum buffer.

From the very beginning, one of our main goals was: Typing math should be as natural as typing plain text. No syntax, no coding—just start typing, no learning required

I just wrote my first blog post in four years! It is called "Deriving Muon". It covers the theory that led to Muon and how, for me, Muon is a meaningful example of theory leading practice in deep learning (1/11)

New paper - Transformers, but without normalization layers (1/n)

And the winner in the self-tuning ruleset, based on Schedule Free AdamW, demonstrated a new level of effectiveness for completely hyperparameter-free neural network training. Roughly ~10% faster training, compared to a NadamW baseline with well-tuned default hyperparameters.

It's crazy how good DiLoCo is. It's currently the main optimizer for distributed optimization, essentially a free lunch for increasing the critical batch size (beyond which the final loss starts degrading) by a factor of 50.

Apparently, just making phrases such as "by the way" their own tokens can give +4% on lots of different tasks. Tokenizers still seem to be pretty disconnected from how we use them in LLMs, I wonder if we can use evolutionary algorithms or something like that to improve them.

We're shipping the first models in the Llama 4 herd! Llama 4 Scout and Llama 4 Maverick, are our first MoE models, natively multimodal and are our most advanced yet — open-sourced as always. ai.meta.com/blog/llama-4-m…

arxiv.org/abs/2504.05295 muon, scion, and dion now. quick skimming... authors proposed low rank updates version of muon for faster training at distributed setup. key idea is LRA, error feedback mechanism (for extremely low rank scenario).

RL is not all you need, nor attention nor Bayesianism nor free energy minimisation, nor an age of first person experience. Such statements are propaganda. You need thousands of people working hard on data pipelines, scaling infrastructure, HPC, apps with feedback to drive