Faster, better model training by reusing old gradients (>10k steps ago) with negligible extra computation? Count me in. arxiv.org/abs/2409.03137

Ademamix optimizer for jax/pytorch: change one line of code, train your model faster.

#ICLR #TrainLLMBetter Tomorrow, #soup of experts, an #hypernetwork conditioned on a simple description of the test distribution: adaptation without retraining (Modularity workshop Sunday). arxiv.org/abs/2502.01804 Still on today... CRISP Importance Sampling for LLM pretraining.