🎉 The MilaNLP lab is excited to present 15 papers and 1 tutorial at #ACL2025 & workshops! Grateful to all our amazing collaborators, see everyone in Vienna! 🚀

[CL] Large Language Model Hacking: Quantifying the Hidden Risks of Using LLMs for Text Annotation J Baumann, P Röttger, A Urman, A Wendsjö... [Bocconi University & University of Zurich] (2025) arxiv.org/abs/2509.08825

📣New paper: Rigorous AI agent evaluation is much harder than it seems. For the last year, we have been working on infrastructure for fair agent evaluations on challenging benchmarks. Today, we release a paper that condenses our insights from 20,000+ agent rollouts on 9

We’re delighted to welcome Eve Fleisig to our MilaNLP lab as a visiting PhD student! ✨

Cool paper by Eddie Yang, confirming our LLM hacking findings (arxiv.org/pdf/2509.08825): ✓ LLMs are brittle data annotators ✓ Downstream conclusions often flip: *LLM hacking risk* is real! ✓ Bias correction methods can help but have tradeoffs ✓ Use human expert whenever possible

The debate over “LLMs as annotators” feels familiar: excitement, backlash, and anxiety about bad science. My take in a new blogpost is that LLMs don’t break measurement; they expose how fragile it already was. doomscrollingbabel.manoel.xyz/p/labeling-dat…