Self-training to maximize self-confidence improves LLM reasoning, without any extrinsic reward.

Dhruv Batra The story is not black and white as the tweet claims: should one have the same prompts for all models (what papers did) vs tune the prompt separately for each model (what blog says; much harder to reproduce). We already highlighted this in the paper and have a baseline to address

Your bimanual manipulators might need a Robot Neck 🤖🦒 Introducing Vision in Action: Learning Active Perception from Human Demonstrations ViA learns task-specific, active perceptual strategies—such as searching, tracking, and focusing—directly from human demos, enabling robust

1/ Maximizing confidence indeed improves reasoning. We worked with Shashwat Goel, Nikhil Chandak Ameya P. for the past 3 weeks (over a zoom call and many emails!) and revised our evaluations to align with their suggested prompts/parsers/sampling params. This includes changing

Shashwat Goel Nikhil Chandak Ameya P. 2/ We also tried our best to compare against concurrent works in the unsupervised RL space. We find using confidence as a reward performs better than having random rewards (while spurious rewards does indeed improve performance). We also find entropy minimization (reverse KL)

Glad we could together improve the scientific discourse around reasoning. Was great to see the authors reach out and incorporate all our feedback!

🚨 The era of infinite internet data is ending, So we ask: 👉 What’s the right generative modelling objective when data—not compute—is the bottleneck? TL;DR: ▶️Compute-constrained? Train Autoregressive models ▶️Data-constrained? Train Diffusion models Get ready for 🤿 1/n

this is such a cool idea! wonder if we can do something similar for robotics.

In RENT, we showed LLMs can improve without access to answers - by maximizing confidence. In this work, we go further: LLMs can improve without even having the questions. Using self-play, one LLM learns to ask challenging questions, while other LLM uses confidence to solve them

this is a really fun idea for improving language models without having to curate more data from Lili !

Cool work! It’s impressive to see how this approach performs, and I can imagine that with stronger verifiers or more dynamic environments, the boundaries could be pushed even further.

Thrilled to share our latest work on improving LLM reasoning -- without using any real data -- via pure self-play/curiosity. Nostalgic to see the return of circa 2017 ideas from unsupervised RL for robotics/games into LLMs. Check out the thread below! 👇