Mad respect to Tré Cool for his open-source work 🫡

Announcing RecurrentGemma! github.com/google-deepmin… - A 2B model with open weights based on Griffin - Replaces transformer with mix of gated linear recurrences and local attention - Competitive with Gemma-2B on downstream evals - Higher throughput when sampling long sequences

🚨Open Source Drop🚨 Databricks is adopting MegaBlocks, and we're releasing the MegaBlocks integration into LLMFoundry. This is a critical component in our Dbrx training stack, and we're super excited to bring MoE training to the community (1/N)

Please welcome MegaBlocks to the Databricks family! databricks.com/blog/bringing-…

“But to us a “register” is a 16x16 tile of data.” Sounds like you guys might like TPUs 😁 Very fun post to read!

Fun collaboration between Databricks Mosaic Research and PyTorch team! We've been working hard to scale MoEs and PyTorch distributed to thousands of GPUs, and this is a great summary of a lot of the cool things we've added to PyTorch. Quick rundown (1/N)

Awesome work 😁🔥 Great to see some of these details being communicated more widely.

Many of you are excited about H100 attention, so it’s a good time to show you Mosaic GPU: a Python DSL for H100s. The attention example matches FA3 performance, while being only ~200 lines of Python: github.com/google/jax/blo… It's easy to install too! Latest JAX packages have it.

Exciting News from Chatbot Arena! Google DeepMind's new Gemini 1.5 Pro (Experimental 0801) has been tested in Arena for the past week, gathering over 12K community votes. For the first time, Google Gemini has claimed the #1 spot, surpassing GPT-4o/Claude-3.5 with an impressive

oh good now i get to watch dan fight all the anime anons all morning

dr. jack morris unclear tbh, data mix is not specified. probably is the big reason for performance

If you’re going to read one thing, read this. Incredible job by these guys putting this together!

Vertical integration is a good way to push efficiency. It also makes working on these models incredibly fun 😁

lol

🚀 Today we’re releasing FlashOptim: better implementations of Adam, SGD, etc, that compute the same updates but save tons of memory. You can use it right now via `pip install flashoptim`. 🚀 arxiv.org/abs/2602.23349 A bunch of cool ideas make this possible: [1/n]