Very interesting read from Shengyao Zhuang and Xueguang Ma!

With great power comes great responsibility! 🕸️🕸️
🤗 Happy to be a part of the collaboration with Snowflake AI (@vivek7ue, Daniel Campos) alongside Jimmy Lin, Ronak Pradeep towards improving retrieval and RAG benchmarks!

Stay tuned in 2024:
#TREC2024RAG and #BEIRv2

Excited to be working with Nandan Thakur Jimmy Lin alongside the wonderful folk Snowflake including Daniel Campos Vivek Raghunathan et al. and in the coming year on TREC RAG and BEIRv2 among many interesting things to come!

While cohere and Snowflake are busy encoding the large corpora, you have BM25 indexes courtesy of us.

The TREC RAG website (trec-rag.github.io/annoucements/2…) has been updated with links to download the updated corpora and topics + qrels from prior years! More topic details to come soon.

Trust Omar and DSPy for concise and clear code samples of super-effective agents!

Oh and one more thing ...
We have task descriptions out too! As always, we open it up to discussions to get the best version of the track for all of you!

Thanks Jerry Liu, we completely agree! The integration with LlamaIndex 🦙 through RankLLM further empowers practitioners and researchers with accessible, open-source tools for refining search candidates. We're excited to see how this pushes what's achievable with open weights!

Thrilled to see the spotlight on RankLLM and the strides we're making in listwise reranking! Shoutout to Ryan Nguyen (a UW ugrad) for the LlamaIndex 🦙 integration, amplifying our user base! Also, RankLLM is nothing without Sahel Sharifymoghaddam. More to come soon :)

You can always trust Leo to do some much needed analysis!

Diving into the challenge of evaluating RAG in TREC's seasoned arena of IR system evaluation! Just realized that we missed out on calling the track RAGnarök. Not a battle, must remind myself, TREC is not a battle 🙃

RAG is all the RAGe these days, but we (still) don't quite know how to evaluate it properly... This year, we are taking a stab at it in the context of TREC, building on 30+ years of experience in evaluating IR systems. trec-rag.github.io

🧐 Exploring deduped MS MARCO V2 as the corpus of choice for TREC 2024 RAG 🚀. Props to NIST folk for a resolution to near dupes in the original corpus. Potential memorization issues acknowledged. You can trust us for baselines soon 😉 Let's discuss! 🔄

#RAG #IR #TREC_RAG

Negation in search is still hard from Orion Weller et al. Nice to see a benchmark where all models tested struggle, motivates the next gen of search systems! Would be nice to see how listwise models like RankZephyr/RankGPT do on these tasks.

monoT5 & duoT5 have been a great pairing that, to date, is heavily leveraged in state-of-the-art pipelines. We see them in a large proportion of IR papers four years later! What better way to start my PhD under the wise master Rodrigo Nogueira :)

We are on a continual quest to simplify reproducibility. Anserini now allows you to reproduce runs with dense and sparse retrieval models (e.g., on MS MARCO and BEIR) directly from a fatjar, 'installed' via wget. Try it out, let us know what you think! anserini.io

✨ We have a *new* TREC track: RAG (Retrieval Augmented Generation) one of the awesome 14 tracks happening in TREC 2024! ✨

Stay tuned in the summer for the call for participants!

Follow for updates: TREC RAG
Webpage: trec-rag.github.io
#RAG #IR #TREC

The TREC calls are out (trec.nist.gov/pubs/call2024.…)!

We introduce TREC RAG as one of the new tracks to aid in building and evaluating systems for this ever-so-important paradigm. I quickly drafted a website last week - trec-rag.github.io, but stay tuned for more 🌟