🤔 Can simple string-matching metrics like BLEU rival reward models for LLM alignment? 🔍 We show that given access to a reference, BLEU can match reward models in human preference agreement, and even train LLMs competitively with them using GRPO. 🫐 Introducing BLEUBERI:

1/ How can a monolingual English speaker 🇺🇸 decide if a French translation 🇫🇷 is good enough to be shared? Introducing ❓AskQE❓, an #LLM-based Question Generation + Answering framework that detects critical MT errors and provides actionable feedback 🗣️ #ACL2025

Do you like trivia? Can you spot when AI is feeding you BS? Or can you make AIs turn themselves inside out? Then on June 14 at College Park (or June 21 online), we have a competition for you.

Very cool and much needed analysis. The counterintuitive scaling limit is very interesting, it's validating to my personal observations about usage of high order of think tokens for just a moderately complex task.

Today is the (human) deadline to sign up for our Human-Computer trivia competition held on June 14, 2024 in College Park, MD. $150 prize for the team who can answer the most questions with the help of an AI.

We had our first human–computer cooperative AI tournament at the UMD. Key takeaways: 1) computers are getting better at trivia 2) they still suck at calibration 3) our teaming mechanic kept the games competitive and mostly fun (at least that’s what the players said).

The models and code are now public! Models on HF: huggingface.co/collections/Sa… Code: github.com/SalesforceAIRe… Project Page: salesforceairesearch.github.io/SweRank/ If you are interesting in integrating the SweRank models as a plug-in within VS Code, please do reach out! We have more exciting

Woohoo! A must-go talk and a well-deserved work by Nishant to be selected for oral.

While we’re building amazing new human-AI systems, how do we actually know if they work well for people? In our #ACL2025 Findings Paper, we introduce SPHERE, a framework for making evaluations of human-AI systems more transparent and replicable. ✨aclanthology.org/2025.findings-…

This week I'll be attending ACL in Vienna, presenting at poster sessions on Monday and Wednesday at 11am, and then giving a keynote talk at the Table Representation Learning workshop on Thursday. Come say hi!

I’ll be presenting this work in Room 1.62 today! If you're curious about how calibration errors in LLMs can be measured through human calibration, come find me and Eve Fleisig @ ACL 2025! 📍Oral Session 3 - HC: Human-centered NLP 📅Monday, July 28@ 2PM

Linguistic theory tells us that common ground is essential to conversational success. But to what extent is it essential? Can LLMs detect when humans lose common ground in conversation? Our ACL 2025 (Oral) paper explores these questions on real-world data. #ACL2025NLP #ACL2025

I'll also be presenting our paper on using question-answer pairs as a new signal for spotting translation errors 🕵️ Come to talk more about MT evaluation! 📍Poster session (Hall X4, X5) 📆Tuesday (7/29) 4-5:30pm 📝aclanthology.org/2025.findings-…

When questions are poorly posed, how do humans vs. models handle them? Our #ACL2025 paper explores this + introduces a framework for detecting and analyzing poorly-posed information-seeking questions! Joint work with Jordan Boyd-Gräber befindet sich in Wien (ACL2025) & Rachel Rudinger! 🔗 aclanthology.org/2025.acl-long.…