🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Our paper "Function-Space Learning Rates" is on arXiv! We give an efficient way to estimate the magnitude of changes to NN outputs caused by a particular weight update. We analyse optimiser dynamics in function space, and enable hyperparameter transfer with our scheme FLeRM! 🧵👇

Sparse attention is one of the most promising strategies to unlock long-context processing and long generation reasoning in LLMs. We performed the most comprehensive study on training-free sparse attention to date. Here is what we found:

It’s worth checking out, and Jeremy Bernstein even shares the slides

arxiv.org/abs/2505.16932 everyone told me even SVD won't improve loss so I only spent a bit effort on improving the iters. After reading leloy! 's post last year, I had the intuition that greedy "contraction" will be a good solution, but didn't know it's optimal.

Psa that theorists should really be more careful when working in continuous time gf vs gd (cts time neq runtime) . I have similar remarks when working with gd vs sgd (gd steps neq sample complexity) . The worst is when you have a result on gf and interpret it as if it were for

great thoughts

If you cite Muon, I think you should definitely cite SSD (proceedings.mlr.press/v38/carlson15.…) by Volkan Cevher et al. (sorry I can't find the handle of other authors) -- which proposed spectral descent.

Flash Linear Attention (github.com/fla-org/flash-…) will no longer maintain support for the RWKV series (existing code will remain available). Here’s why:

One of the best ways to reduce LLM latency is by fusing all computation and communication into a single GPU megakernel. But writing megakernels by hand is extremely hard. 🚀Introducing Mirage Persistent Kernel (MPK), a compiler that automatically transforms LLMs into optimized

huggingface website is down. plz fix it Hugging Face

~8/8~ We release our NSA kernel for experimentation and research here: github.com/tilde-research… At Tilde, we believe interpretability is the path towards building better models. If that sounds cool, reach out!

Join our ML Theory group next week as they welcome Tony S.F. on July 3rd for a presentation on "Training neural networks at any scale" Thanks to Andrej Jovanović Anier Velasco Sotomayor and Thang Chu for organizing this session 👏 Learn more: cohere.com/events/Cohere-…

I ran the same watermark test against Adam a few weeks back. This is the result I got:

finbarr You can fix it with doing the softmax in fp32 arxiv.org/abs/2506.13585

It’s not a bug if you use it right! Some work use similar property to accelerate model inferencing, e.g. SliceGPT and LaRoSa (except they use orthogonal matrices to avoid P inverse here).

They got k2 in contributor list of k2 report lol

very insightful

Looks like extremely exciting and useful work by Kwangjun Ahn, Byron Xu, Natalie Abreu, John Langford and Gagik Magakyan github.com/microsoft/dion/ (2/2)