Check out the paper: dl.acm.org/doi/10.1145/36…

Goodbye Washington! We had a fantastic week with interesting talks, discussions, and new ideas at #SIGIR24 #SIGIR2024. We hope to see you all again next year in Italy :)

Jan Heinrich Merker presented our short paper at #ArgMining2024 at #ACL2024 today. We proposed to add "semantics" to lexical retrieval models for argument retrieval. w/ Matthias Hagen, Maik Fröbe, Danik Hollatz. Paper: aclanthology.org/2024.argmining…

Great presentation about RAG in biomedical domain Jan Heinrich Merker #CLEF2024 #BIOASQ2024

alexander bondarenko Thanks! You can find out paper and code online: webis.de/publications.h… github.com/webis-de/clef2… #CLEF2024

Maik Fröbe Great talk on the Team OpenWebSearch.eu @[email protected]'s submission to the QuantumCLEF shared task on how to exploit #QuantumComputing for feature selection in IR 👍 Exciting new direction of IR research at #CLEF2024 #QuantumCLEF2024

Follow-up on our #BIOASQ2024 submission: We actually submitted the best approach for some of the tasks 👍 Looking forward to further improving Medical RAG! #CLEF2024

Lightning IR: Straightforward Fine-tuning and Inference of Transformer-based Language Models for Information Retrieval Introduces a PyTorch Lightning-based framework for fine-tuning and inference of transformer models in IR. 📝arxiv.org/abs/2411.04677 github.com/webis-de/light…

Happy to share our framework for fine-tuning and running neural ranking models, Lightning IR, was accepted as a demo at #WSDM25 🥳 Pre-print: arxiv.org/abs/2411.04677 Code: github.com/webis-de/light… Docs: webis.de/lightning-ir A quick rundown of Lightning IR's main features:

Next up at #ECIR2025, Maik Fröbe presenting his fantastic work on corpus sub sampling and how to more efficiently evaluate retrieval systems

📢 Our paper "The Viability of Crowdsourcing for RAG Evaluation" has been accepted to #SIGIR2025 ! We compared how good humans and LLMs are at writing and judging RAG responses, assembling 1800+ responses across 3 styles, and 47K+ pairwise judgments in 7 quality dimensions. 🧵➡️

🧵2/4 Key findings: 1️⃣ Humans write best? No! LLM responses are rated better than human. 2️⃣ Essay answers? No! Bullet lists are often preferred. 3️⃣ BLEU? No! Reference-based metrics don't align with human preferences. 4️⃣ LLMs as judges? No! Prompted models label inconsistently.

🧵 3/4 This fundamentally challenges previous assumptions about RAG evaluation and system design. But we also show how crowdsourcing offers a viable and scalable alternative! Check out the paper for more. 📝 Preprint @ downloads.webis.de/publications/p…⚙️Code/Data is openly available.

🧵 4/4 Credit and thanks to the author team Lukas Gienapp, Tim Hagen, Maik Fröbe, Matthias Hagen, Benno Stein, Martin Potthast, and Harry Scells – you can also catch some of them at #ECIR2025 currently if you want to chat about RAG!

What an honor to receive both the best short paper award and the best paper honourable mention award at #ECIR2025. Thank you to all the co-authors Maik Fröbe Harry Scells Shengyao Zhuang Bevan Koopman Guido Zuccon Benno Stein Martin Potthast Matthias Hagen 🥳

Maik Fröbe Harry Scells Shengyao Zhuang Bevan Koopman Guido Zuccon Benno Stein Martin Potthast Matthias Hagen Short: Rank-DistiLLM: Closing the Effectiveness Gap Between Cross-Encoders and LLMs for Passage Re-ranking webis.de/publications.h… Full: Set-Encoder: Permutation-Invariant Inter-Passage Attention for Listwise Passage Re-Ranking with Cross-Encoders webis.de/publications.h…

Do not forget to participate in the #TREC2025 Tip-of-the-Tongue (ToT) Track :) The corpus and baselines (with run files) are now available and easily accessible via the ir_datasets API and the HuggingFace Datasets API. More details are available at: trec-tot.github.io/guidelines

Want to know how to make bi-encoders more than 3x faster with a new backbone encoder model? Check out our talk on the Token-Independent Text Encoder (TITE) #SIGIR2025 in the efficiency track. It pools vectors within the model to improve efficiency dl.acm.org/doi/10.1145/37…

Happy to share that our paper "The Viability of Crowdsourcing for RAG Evaluation" received the Best Paper Honourable Mention at #SIGIR2025! Very grateful to the community for recognizing our work on improving RAG evaluation. 📄 webis.de/publications.h…

Honored to win the ICTIR Best Paper Honorable Mention Award for "Axioms for Retrieval-Augmented Generation"! Our new axioms are integrated with ir_axioms: github.com/webis-de/ir_ax… Nice to see axiomatic IR gaining momentum.