Online interaction is probably a defining property of RL. But with the rise of offline algo, it is not clear if the “online” bit of RL is necessary for RLHF. We hypothesis test the causes of the perf gap between online and offline alignment. arxiv.org/pdf/2405.08448… Details in🧵

Two-stage RLHF or one-stage DPO: Which one is better for learning from preferences? Equal under strong assumptions, but representation differences break the tie. Our paper reveals their fine-grained performance gaps under various conditions. paper: arxiv.org/abs/2505.19770

Google DeepMind just put out a Lean 4 repo of formalized open math problems. We need more contributions to the database! github.com/google-deepmin…

[LG] Why Gradients Rapidly Increase Near the End of Training A Defazio [FAIR at Meta] (2025) arxiv.org/abs/2506.02285

🚨 New Paper 🧵 How effectively do reasoning models reevaluate their thought? We find that: - Models excel at identifying unhelpful thoughts but struggle to recover from them - Smaller models can be more robust - Self-reevaluation ability is far from true meta-cognitive awareness

[LG] Probably Approximately Correct Labels E J. Candès, A Ilyas, T Zrnic [Stanford University] (2025) arxiv.org/abs/2506.10908

How does the loss of external context affect a language model’s learning ability to ground its responses? Check out our latest work led by hyunji amy lee, where we introduce CINGS, a simple training method that significantly improves grounding in both text and vision-language

🚀 GR00T Dreams code is live! NVIDIA GEAR Lab's open-source solution for robotics data via video world models. Fine-tune on any robot, generate 'dreams', extract actions with IDM, and train visuomotor policies with LeRobot datasets (GR00T N1.5, SmolVLA). github.com/NVIDIA/GR00T-D…

🤔 How do we train AI models that surpass their teachers? 🚨 In #COLM2025: ✨Delta learning ✨makes LLM post-training cheap and easy – with only weak data, we beat open 8B SOTA 🤯 The secret? Learn from the *differences* in weak data pairs! 📜 arxiv.org/abs/2507.06187 🧵 below

⁉️Why do reward models suffer from over-optimization in RLHF? We revisit how representations are learned during reward modeling, revealing “hidden state dispersion” as the key, with a simple fix! 🧵 Meet us at ICML Conference! 📅July 16th (Wed) 11AM–1:30PM 📍East Hall A-B E-2608

Our paper "Do Large Language Models Perform Latent Multi-Hop Reasoning without exploiting shortcuts?" will be presented at #ACL2025 today. 📍 Mon 18:00-19:30 Findings Posters (Hall X4 X5) Please visit our poster if you are interested! ✨

🎉🎉 Super pround to share that our work is accepted to #NeurIPS2025 !! Huge thanks to all the co-authors. 👏👏

🧐 LLMs aren’t great at judging their own correctness. ❗But history across models helps! We present Generalized Correctness Models (GCMs), which learn to predict correctness based on history, outperforming model-specific correctness and larger models' self-confidence.