Standard attention is softmax(QKᵀ)V. We explore 2-simplicial attention using additional keys K' for richer interactions: softmax(Q(K⊗K')ᵀ). 🧵

Excited to share that I’ll be presenting two oral papers in this ICML—see u guys in Vancouver!!🇨🇦 1️⃣ arxiv.org/abs/2502.06768 Understanding Masked Diffusion Models theoretically/scientifically 2️⃣ arxiv.org/abs/2502.09376 Theoretical analysis on LoRA training

Announcing Ambient Protein Diffusion, a state-of-the-art 17M-params generative model for protein structures. Diversity improves by 91% and designability by 26% over previous 200M SOTA model for long proteins. The trick? Treat low pLDDT AlphaFold predictions as low-quality data

Incredible to see how fast the field moves since we worked on masked diffusion arxiv.org/abs/2406.04329. Huge congratulations to Jaeyeon (Jay) Kim Sitan Chen for the award!

Thrilled to share that our work received the Outstanding Paper Award at ICML! I will be giving the oral presentation on Tuesday at 4:15 PM. Jaeyeon (Jay) Kim @ICML and I both will be at the poster session shortly after the oral presentation. Please attend if possible!

A team from #KempnerInstitute, Harvard SEAS & Computer Science at UT Austin has won a best paper award at #ICML2025 for work unlocking the potential of masked diffusion models. Congrats to Jaeyeon (Jay) Kim @ICML, Kulin Shah, Vasilis Kontonis, Sham Kakade and Sitan Chen. kempnerinstitute.harvard.edu/news/kempner-i… #AI

congrats on the award!! great to see more work designing insightful tasks that bring out the role of token ordering & difficulty (should i say "indecipherability" ;-) ). I think the idea of learning token-level subproblems is broken both for diffusion (and next-token learning)

Today Chen Wu and I will be presenting our #ICML work on creativity in the Oral 3A Reasoning session (West Exhibition Hall C) 10 - 11 am PT Or please stop by our poster right after @ East Exhibition Hall A-B #E-2505 11am-1:30pm. (Hope you enjoy some silly human drawings!)

Congrats to the Kim, Shah, et al. team from Harvard on winning an outstanding paper award at ICML for their work on understanding generation orderings in masked diffusion! And thank you for the shout-out to @InceptionAILabs!

📢 Thrilled to share our new paper, LASER: Attention with Exponential Transformation, accepted at ICML2025, work done at Google. Come by our poster presentation! 🗓️ Thurs, July 17th, 4:30-7pm 📍 West Exhibition Hall B2-B3, # W-915 Read the full paper here: arxiv.org/abs/2411.03493

Congrats to UT Austin students Kulin Shah and Vasilis Kontonis who won an Outstanding Paper Award at #ICML2025! Their work pushes the boundaries of how AI models learn and understand the world. We’re proud to see @utcompsci students leading at the frontier of machine learning.

Excited to announce the Reliable ML workshop at NeurIPS Conference 2025! 📣 How do we build trustworthy models under distribution shift, adversarial attacks, strategic behavior, and missing data? Fantastic lineup of speakers/panelists: Ahmad Beirami (Ahmad Beirami), Surbhi Goel

I am excited to announce that our AI institute (Institute for Foundations of Machine Learning, IFML) has been renewed. IFML was part of the first cohort of AI Institutes announced in 2020. Led by UT Austin, the new award will build on the trajectory of the past five years and

Wrote up some notes providing an introduction to discrete diffusion models, going into the theory of time-inhomogeneous CTMCs via generators/time-evolution systems. What motivated me was the sheer difficulty of finding a useful reference which laid out the theory (e.g.

We’re releasing Open-dCoder — the first fully open diffusion large language model for code: 🏋️ Pretraining pipeline 📂 Training data 📊 Evaluation suite ⚡ Inference code 📦 Checkpoints 🔗 Repo: github.com/pengzhangzhi/O… 📖 Blog: bit.ly/oDLLM-blog (1/n)

Mercury Coder now supports Apply-Edit capabilities, providing quality on par with GPT-5 at speeds 46x faster!

🚨 New paper! We introduce a planner-aware training tweak to diffusion language models. ⚡ One-line-of-code change to the loss 💡 Fixes training–inference mismatch 📈 Strong gains in protein, text, and code generation arxiv.org/abs/2509.23405 (1/n)

Proud of our GPT-25 work (Giannoulis-Pantis-Tzamos) teaching Transformers to solve Combinatorial Problems in NeurIPS Conference. Even simple Sudokus stump the most capable LLMs without external tools; we go from 0% to 99.8% accuracy teaching them to reason and search like humans!

Check out this new work on techniques for constructing Transformers for algorithmic tasks! Excited to have been part of this project!

Diffusion #LLMs, our work provides - Interesting insights into its working - test-time scaling - can outperform the fine-tuned GRPO version - raises interesting questions about optimal inference in dLLMs (a lot to gain and explore) x.com/amritsinghbedi…