🚨 How do attention sinks relate to information flow in LLMs? We show how massive activations create attention sinks and compression valleys, revealing a three-stage theory of information flow in LLMs. 🧵 w/ Enrique* Federico Barbero xiaowen dong Michael Bronstein Yann LeCun Ravid Shwartz Ziv

#AITHYRA, Vienna's new Biomedical AI institute, is hiring Postdocs! Come work with us. Openings in: 🔹 Generative AI 🔹 Multimodal ML 🔹 Virology 🔹 Enzyme Function Apply by Nov 20: oeaw.ac.at/aithyra/postdo… #PostDoc #AI #ML #Vienna #ScienceJobs

Introducing Generalised Flow Maps 🎉 A stable, few-step generative model on Riemannian manifolds 🪩 📚 Read it at: arxiv.org/abs/2510.21608 💾 Code: github.com/olsdavis/gfm Michael Albergo Nicholas Boffi Michael Bronstein Joey Bose

Cool news: our extended Riemannian Gaussian VFM paper is out! 🔮 We define and study a variational objective for probability flows 🌀 on manifolds with closed-form geodesics. Floor Eijkelboom Alison Cong Liu Max Welling Jan-Willem van de Meent Erik Bekkers 🔥 📜 arxiv.org/abs/2502.12981

We figured out flow matching over states that change dimension. With "Branching Flows", the model decides how big things must be! This works wherever flow matching works, with discrete, continuous, and manifold states. We think this will unlock some genuinely new capabilities.

Flow Matching models often struggle to balance memorization and generalization. 😱 We set out to fix this — by using the geometry of the data manifold. Introducing Carré du Champ Flow Matching (CDCFM)🧑‍🎨🥖 — improving generalization without sacrificing sample quality.

Learning diffusion geometry from Iolo Jones was a highlight of this project for me. Highly recommend checking out his work: General diffusion geometry: arxiv.org/abs/2405.10858 Application to manifolds/differential geometry: arxiv.org/abs/2411.04100

TIL Flow matching faces a quality-generalisation trade-off that can be mitigated working with the (empirical) data manifold directly! 🇪🇺

We also found that flow matching (FM) can overfit unevenly across the data manifold. 🧐 In this heterogeneous toy example (large + small circle), the large circle is memorized early while the other is learned much later. CDCFM mitigates this spatial imbalance. 🔥