🔥Telegram İfşa

kexue.fm/archives/11196 This series opener explores the steepest-descent direction for equality-constrained optimization, starting with an SGD variant tailored to the hypersphere constraint.

kexue.fm/archives/11215 Muon on orthogonal manifold, first from Jeremy Bernstein

Muon + Stiefiel kexue.fm/archives/11221 solved the open problem at docs.modula.systems/algorithms/man… from Jeremy Bernstein . cc leloy!

A fun fact: Adam remains the dominant optimizer today, yet even it has had only scant opportunities to be verified on trillion-parameter models; Muon, proposed less than a year ago, has already trained at that scale.

Muon + Spectral Sphere: kexue.fm/archives/11241

Cool Papers + Zotero kexue.fm/archives/11250 papers.cool

Rethinking the Relationship Between Learning Rate and Batch Size (Part I): Current Status kexue.fm/archives/11260

Why is Adam's Update RMS 0.2? kexue.fm/archives/11267 TLDR: Adam_Update_RMS ≈ sqrt((1 - beta1) / (1 + beta1))

Rethinking the Relationship Between Learning Rate and Batch Size (Part II): Mean Field kexue.fm/archives/11280

Rethinking the Relationship Between Learning Rate and Batch Size (Part III): Muon kexue.fm/archives/11285

Rethinking the Relationship Between Learning Rate and Batch Size (Part IV): EMA kexue.fm/archives/11301

Asymptotic Estimate of Weight RMS for AdamW kexue.fm/archives/11307

Why does linear attention need Short Conv? kexue.fm/archives/11320

DiVeQ: A Very Concise Training Method for VQ kexue.fm/archives/11328

Fast (but non-rigorous) estimation of the spectral norm of random matrices kexue.fm/archives/11335

Beyond MuP: 1. The Self-Cultivation of a Good Model kexue.fm/archives/11340 This series will share some top-down attempts at model optimization -- an extension and expansion of the ideas of MuP and Muon.

Low-precision attention may suffer from biased rounding errors kexue.fm/archives/11371

Dual Gradient Descent kexue.fm/archives/11388 The method learned from Jeremy Bernstein Jeremy Bernstein

Expectation of the maximum of gaussian random variables kexue.fm/archives/11390