How do you know if a method is better, or just has better hyperparameters? He He, Kyunghyun Cho, and I give a new tool to answer this in our #NAACL2024 paper: "Show Your Work with Confidence" arxiv.org/abs/2311.09480. Use it in your own work with just a "pip install opda"! 🧵 1/8

Tal Linzen Ekin Akyürek Yoav Artzi Neel Nanda I really like the paper from Jane Pan (w Danqi Chen) abt this: arxiv.org/abs/2305.09731. ICL in big models is clearly a mix of task recognition and "real learning" (you're not learning to translate from 3 examples, but you're not getting an arbitrary label mapping from the prior)

Do reasoning models know when their answers are right?🤔 Really excited about this work led by Anqi and Yulin Chen. Check out this thread below!

We're excited to receive wide attention from the community—thank you for your support! We release code, trained probes, and the generated CoT data👇 github.com/AngelaZZZ-611/… We have labeled answer data on its way. Stay tuned!

What does it mean for #LLM output to be novel? In work w/ John(Yueh-Han) Chen, Jane Pan, Valerie Chen, He He we argue it needs to be both original and high quality. While prompting tricks trade one for the other, better models (scaling/post-training) can shift the novelty frontier 🧵

I'll be at ACL Vienna 🇦🇹 next week presenting this work! If you're around, come say hi on Monday (7/28) from 18:00–19:30 in Hall 4/5. Would love to chat about code model benchmarks 🧠, simulating user interactions 🤝, and human-centered NLP in general!

Bored of seeing pristine, perfect posters? Come see me at Hall X5, Board 105 at 6pm to witness my masterpiece, featuring bonus Sharpie scribbles and a QR code that betrayed me at the last moment 😤