Simply adding Gaussian noise to LLMs (one step—no iterations, no learning rate, no gradients) and ensembling them can achieve performance comparable to or even better than standard GRPO/PPO on math reasoning, coding, writing, and chemistry tasks. We call this algorithm RandOpt.

I made an interactive blog post about how JPEG image compression works: sophielwang.com/blog/jpeg

Counterintuitive? Longer sequences have more repeated signals and more similar to overfitting? Is that the picture?

Arrived at APS March meeting 🤩 Will talk about Neural Scaling Laws Trilogy (Mar 19, at 🚨Physics of Learning and Adaptation III)! Happy to schedule chats with physicists interested in LLMs‼️

Similar to Fig. 20 and 24 in our arxiv.org/abs/2602.05970. When transition is smooth ODE, dense connection can be much better. When transition is SDE (or high-order derivatives don’t exist), there is no big change. Actually we found LLMs are more like SDE.

0.148 agrees with what I said.

Seems to be a systematic scaling study for diffusion language model 👍 Not surprised that the exponents are still similar to Chinchilla. But the origin of 21.8x speedup? So far I can imagine diffusion models enable better hyperparameter choices.

One obvious reason may be for larger batch size, there are fewer steps.

How should I interpret the exponent 1/4? The best case of lower bound? max(min(alpha_D))?

Some works also reported that optimal batch size scales as dataset size^1/3. I wonder if these are related to my 1/3 scaling, arxiv.org/abs/2602.03685, by balancing that loss and an extra loss due to gradient noise. (My theory studied perfect gradients).

Can’t agree more 😂 or please report the fitting standard error…