We simply explain and illustrate Mamba and (Selective) State Space Models – SSMs. 📺 youtu.be/vrF3MtGwD0Y SSMs match performance of transformers, but are faster and more memory-efficient than them. This is crucial for long sequences! Incredible work by Albert Gu Tri Dao! 👏

Lives Ended in Gaza… nytimes.com/interactive/20…

Our 12 scaling laws (for LLM knowledge capacity) are out: arxiv.org/abs/2404.05405. Took me 4mos to submit 50,000 jobs; took Meta 1mo for legal review; FAIR sponsored 4,200,000 GPU hrs. Hope this is a new direction to study scaling laws + help practitioners make informed decisions

🎨Spent some time refactoring the 2021 post on diffusion model with new content: lilianweng.github.io/posts/2021-07-… ⬇️ ⬇️ ⬇️ 🎬Then another short piece on diffusion video models: lilianweng.github.io/posts/2024-04-… (Yes, I had an intensive weekend🥹)

In the long run we are all dead -Keynes

Exciting update -- Llama-3 full result is out, now reaching top-5 on the Arena leaderboard🔥 We've got stable enough CIs with over 12K votes. No question now Llama-3 70B is the new king of open model. Its powerful 8B variant has also surpassed many larger-size models. What an

Young scientists regularly ask me for career advice. Academia or industry? Big company or startup? US or Europe?  Good scientists in AI disciplines are fortunate to have many choices. But choosing can be stressful. I always give the same advice. 1/10

We’ll be streaming live on openai.com at 10AM PT Monday, May 13 to demo some ChatGPT and GPT-4 updates.

😢

I'm excited about this! Our team has been working really hard to improve Gemini 1.5 capabilities significantly on multiple fronts and in particular MATH/STEM! Please see the report here: goo.gle/GeminiV1-5

With Albert Gu, we’ve built a rich theoretical framework of state-space duality, showing that many linear attn variants and SSMs are equivalent! The resulting model, Mamba-2 is better & faster than Mamba-1, and still matching strong Transformer arch on language modeling. 1/

Brilliant work by the Android agents team at Google DeepMind 📌 The authors introduce ANDROIDCONTROL, a new dataset of 15,283 human demonstrations of everyday tasks across 833 Android apps. Each task includes both high-level and low-level instructions. This allows studying agent

For many years as a Stanford NLP Group PhD student, I loved attending these seminars. It’s good to be back, this time as a guest speaker! I’ll discuss my group’s recent progress on understanding and auditing large language models

way back in 2022, the best model in the world was text-davinci-003. it was much, much worse than this new model. it cost 100x more.

New fundraiser to support 25 Nigerian students to attend Deep Learning Indaba in September! 🌍 In 2022 we supported 8 Nigerian students to attend Indaba. This year we are raising $20k to support 25(!) of them to travel to Senegal for likely the most important career event in their lives!

Incredibly honored and humbled by the overwhelming response to my tutorial, and thank you everyone who attended in person. Truly heartwarming to hear how much you enjoyed it. Many have been asking for a recording, and I prepared one with my own subtitles youtu.be/yBL7J0kgldU

Bad news (1/2): video taken down by ICML ([email protected]) for copyright. While I can't agree (the consent I signed allows me to publish elsewhere) - I will respect it to save time for more important things. To bad I delayed many things and spent 20+ hrs preparing the video.

🥲🥲

I'm excited to announce that I am joining the OpenAI Board of Directors. I'm looking forward to sharing my perspectives and expertise on AI safety and robustness to help guide the amazing work being done at OpenAI.

I'll be presenting our work on investigating the role of meta-prompt components in automatic prompt engineering, i.e., "𝗽𝗿𝗼𝗺𝗽𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿𝗶𝗻𝗴 𝗮 𝗽𝗿𝗼𝗺𝗽𝘁 𝗲𝗻𝗴𝗶𝗻𝗲𝗲𝗿 ", at Findings Session 1 (Mon 12:45) and NLRSE Workshop (Thu 4pm)! Please come say hi! 👋