🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Our 12 scaling laws (for LLM knowledge capacity) are out: arxiv.org/abs/2404.05405. Took me 4mos to submit 50,000 jobs; took Meta 1mo for legal review; FAIR sponsored 4,200,000 GPU hrs. Hope this is a new direction to study scaling laws + help practitioners make informed decisions

Our recent paper shows: 1. Crrent LLM safety alignment is only a few tokens deep. 2. Deepening the safety alignment can make it more robust against multiple jailbreak attacks. 3. Protecting initial token positions can make the alignment more robust against fine-tuning attacks.

1/ LLMs are often used to generate text new math questions. But can they generate challenging math questions? Current methods yield Qs that're either easy or too similar to existing questions. Our new paper "AI-Assisted Generation of Difficult Math Questions" shows how to

💡 The Mathematics of Modern Machine Learning (M3L) workshop is back for its 2nd edition at NeurIPS 2024. Submit your work and share your perspectives on modern ML theory! 📅 Submission ddl: Sept 29, 2024 (2 days after ICLR abstract ddl) 🌐 sites.google.com/view/m3l-2024

We've extended the #M3L submission deadline to October 1st AoE to align with ICLR timelines. We look forward to your work!

Thanks to everyone who joined and supported the M3L workshop this year! It was so exciting to see so many inspiring ideas and discussions. Unfortunately, I got a fever one day before the workshop and couldn’t attend in person. Looking forward to seeing you all next year!

Kids use open textbooks for homework. Can LLM training benefit from "helpful textbooks" in context with no gradients computed on these tokens? We call this Context-Enhanced Learning – it can exponentially accelerate training while avoiding verbatim memorization of “textbooks”!

🚨Ever wonder why diffusion models generate nonsensical text? Our latest study at #ICLR2025 uncovers "Local Generation Bias"—a hidden training bias causing textual hallucinations! 🧠 Key finding: Diffusion models independently generate symbols locally without global context.

The success of RLHF depends heavily on the quality of the reward model (RM), but how should we measure this quality? 📰 We study what makes a good RM from an optimization perspective. Among other results, we formalize why more accurate RMs are not necessarily better teachers! 🧵

We will present this paper at #ICLR2025! 1. 𝐎𝐫𝐚𝐥 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟏𝐃 (𝐓𝐡𝐮𝐫𝐬𝐝𝐚𝐲 𝟏𝟎:𝟒𝟐𝐚𝐦) Ashwinee Panda will give a talk 2. 𝐏𝐨𝐬𝐭𝐞𝐫 𝐒𝐞𝐬𝐬𝐢𝐨𝐧 𝟒 (𝐅𝐫𝐢𝐝𝐚𝐲 𝟑𝐩𝐦) Come to chat with Ashwinee Panda Kaifeng Lyu Xiao Ma Ahmad Beirami Unfortunately, I

Thrilled to share that our paper “Safety Alignment Should be Made More Than Just a Few Tokens Deep” has received an ICLR 2025 Outstanding Paper Award! This project began as an effort to defend against fine-tuning attacks with constrained supervised fine-tuning (SFT). Along the

What's the optimal learning rate schedule for LLM pretraining? Come meet us this afternoon! Poster Presentation: 🗓 Friday, April 25 🕒 3:00 PM – 5:30 PM CST 📍 Hall 3 + Hall 2B, Poster #237

Excited to present our paper this morning at ICLR 2025, revealing the gap in CoT reasoning between RNNs and Transformers! Poster Presentation: 🗓 Saturday, April 26 📷 10:00 AM – 12:30 PM 📍 Hall 2, Poster #640

Excited to share our new method ✏️PENCIL! It decouples space complexity from time complexity in LLM reasoning, by allowing model to recursively erase and generate thoughts. Joint work w. my student Chenxiao Yang , along with Nati Srebro Bartom and David McAllester.