Submission deadline extended- August 23rd! Looking forward to reading your works.

Keynote speech is supposed to be technically sound and enlightening. For students who feel discouraged / unwelcome by any speech - simply ignore, especially when you actually know so much more about what’s happening than the speakers.

Excited to have presented our TACL paper about the sensitivity of LLMs to prompt paraphrases at #ACL2024! 📊✨ Guy Kaplan, Dan H.M 🎗, Rotem Dror, Hyadata Lab (Dafna Shahaf), Gabriel Stanovsky direct.mit.edu/tacl/article/d…

I am looking for NLP/CV datasets that: 1. include annotator IDs 2. have multiple annotators per item 3. feature interesting tasks (including generation) Any suggestions?

For a new and exciting research about data we need your help! 🤓📑💻 If your dataset meets these criteria, we would love to cite your research and analyze the data - exciting results are guaranteed:) #DataScience #NLP #vision Journal of Data-centric Machine Learning Research

Do you think LLMs could win a Nobel Prize one day? 🤔 Can NLP predict heroin addiction outcomes, uncover suicide risks, or simulate brain activity? 🧠 Lotem Peled, Roi Reichart, and I wrote a blog post about NLP for Human-Centric Sciences 🤖👩‍🔬 👇👇👇 nitaytech.github.io/blog/2024/nlp4…

In our new preprint, Roi Reichart ,Rotem Dror and I propose a new statistical procedure: The Alternative Annotator Test (alt-test) The goal? To help researchers justify using LLMs over humans—if the LLM passes, its annotations can be confidently trusted😎 arxiv.org/abs/2501.10970

Check out our new pre-print on statistically sound methodology to verify the quality of LLM-as-a-judge annotation. Nitay Calderon Rotem Dror

A few months ago I told you that I'm working on something awesome and I need some datasets...well here is the first output of that effort and I sincerely think it's amazing 🤩 check it out! Nitay Calderon Roi Reichart #llm-as-a-judge, #nlpevaluation

Preferences drive modern LLM research and development: from model alignment to evaluation. But how well do we understand them? Excited to share our new preprint: Multi-domain Explainability of Preferences arxiv.org/abs/2505.20088 Roi Reichart Liat 🧵👇 1/11

🔥הסקירות ממשיכות לזרום ל-X🔥 🧵 המאמר היומי של מייק: 25.06.25 The Alternative Annotator Test for LLM-as-a-Judge: How to Statistically Justify Replacing Human Annotators with LLMs מאמר 🇮🇱 תפנית מעניינת מתרחשת בתקופה האחרונה בעולם של הערכת ביצועי מודלים. אנחנו כבר לא שואלים רק

Everyone uses LLMs to annotate data or evaluate models in their research. But how can we convince others (readers, collaborators, reviewers!!!) that LLMs are reliable? 🤖 Here’s a simple (and low-effort) solution: show the LLM is a *comparable alternative annotator* ✅

The Alternative Annotator Test (alt-test) is a new statistical procedure proposed in our ACL 2025 paper! 🇦🇹🇦🇹 Rotem Dror Roi Reichart The goal? To help justify using LLMs over humans. If the LLM passes the test, its annotations can be trusted 😎 arxiv.org/abs/2501.10970

Tomorrow morning (9AM🌅) I'll be giving a keynote talk at LAW (linguistic annotation workshop) at #ACL2025 on annotations in the era of LLMs - see you there!!🌟🌟

A strange trend I've noticed at #ACL2025 is that people are hesitant to reach out to papers/"academic products" authors. This is unfortunate for both parties! A simple email can save a lot of time to the sender, but is also one of my favorite kind of email as the receiver!