✨What do language models know about grammar? In our new TACL paper we approach this question using the Structural Priming paradigm, that has been used to uncover humans' comprehension of abstract syntax. 📜Blog resources.illc.uva.nl/illc-blog/prob… 📑Paper arxiv.org/pdf/2109.14989… 👇Thread

Very excited to give a seminar tomorrow on the ethics of Large Language Models at UvA Amsterdam! Be sure to stop by if you are around, or ping me if you'd like to watch the live streaming on Zoom. Big thanks to Sandro Pezzelle for the kind invitation. I am looking forward!

Just published: New research on how language technologies that perpetuate stereotypes actively cement social hierarchies. 'Which Stereotypes Are Moderated and Under-Moderated in Search Engine Autocompletion?' together with Alina Leidinger, dl.acm.org/doi/10.1145/35… vera.ai

First few days at #EMNLP2023 have been a blast! Tomorrow, I'll be presenting our poster on robustness in LLM evaluation (session 1): arxiv.org/abs/2311.01967 Come say hi if you wanna chat about robustness/safety of LLMs!

At the #EMNLP2023 poster session right now, all the way in the back: Check out Alina Leidinger's poster on the linguistic properties of successful prompts and start treating them as hyperparameters to be re-tuned for every task and model!

New Article: "Undesirable Biases in NLP: Addressing Challenges of Measurement" by van der Wal, Bachmann, Leidinger, van Maanen, Zuidema, and Schulz jair.org/index.php/jair…

Our paper on bias in NLP x psychometrics has been published in JAIR! 🎉

"The examples might be surprising, but the broad strokes of the research aren’t. It’s well established at this point that all models contain biases, albeit some more egregious than others." “We call on researchers to rigorously test their models for the cultural visions they

How open text-analyzing models respond to questions relating to LGBTQ+ rights, social welfare, surrogacy and more? Giada Pistilli Alina Leidinger Atoosa Kasirzadeh Yacine Jernite Sasha Luccioni, PhD 🦋🌎✨🤗 MMitchell found that they tend to answer questions inconsistently, which reflects biases embedded

🧵 Very excited to introduce our latest research: "CIVICS: Building a Dataset for Examining Culturally-Informed Values in Large Language Models." Access the pre-print here: huggingface.co/papers/2405.13…

Our paper has been accepted at AI, Ethics, and Society Conference (AIES)! I am on maternity leave and won't be able to attend, but be sure to get a chance to talk about it with my co-authors Alina Leidinger, MMitchell, Atoosa Kasirzadeh Yacine Jernite, Sasha Luccioni, PhD 🦋🌎✨🤗 ✨

Announcing NeurIPS Workshop: EvalEval 2024! 🚀 As generative AI rapidly transforms our world, a critical question looms: How do we measure and evaluate its broader societal impacts? 📄 Our recent collaborative paper (arxiv.org/pdf/2306.05949) reveals a lack of standardized

Working on #bias & #discrimination in #NLP? Passionate about integrating insights from other disciplines? Want to discuss current limitations of #LLM bias mitigation? 👋Join the workshop New Perspectives on Bias and Discrimination in Language Technology; 4&5 Nov in #Amsterdam!

Poster presentation today at #ACL2024NLP at 10:30 🥳🥳 Come say hi if you want to chat about robust evaluation! Paper 👇 aclanthology.org/2024.acl-short…

We're nearly 2 weeks away from the deadline for Tiny Papers for our workshop on social impact evaluation of genAI. If you have thoughts, critiques, WIP, or resources on that topic, now's the time to make them a quick 2-pager! evaleval.github.io/call-for-paper…

📖For our weekly MilaNLP lab seminar, it was a pleasure to have Alina Leidinger presenting "How are LLMs mitigating stereotyping harms." #NLProc

Very happy to be presenting this project at AI, Ethics, and Society Conference (AIES) today! If you're at #AIES2024, drop by our poster at 6 p.m. today!