🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Can code LLMs keep up with changes in APIs? We've previously studied updating facts in LLMs, and this project advances that research into more complex domains!

Our paper has been accepted by Conference on Language Modeling🎉! Our analysis reveals behaviors of LM when generating long-form answers with retrieval augmentation, and provides directions for future work in this line!

Accepted at Conference on Language Modeling with scores of 9/8/7/6 🎉 We show current LMs struggle to handle multiple documents featuring confusing entities. See you in Philadelphia!

🥝KIWI at #ACL2024 : Check out our poster and talk to my collaborators Kyle Lo Luca Soldaini 🎀 !

VLMs can generate long-form answers to visual questions (LFVQA). What information do these long-form answers contain? How can we evaluate them? In our #COLM2024 paper, we introduce VizWiz-LF, a dataset of long-form answers to visual questions from blind and low vision people.

Tomorrow is the day! We cannot wait to see you at #ACL2024 ACL 2025 Knowledgeable LMs workshop! Super excited for keynotes by Peter Clark Luke Zettlemoyer Tatsunori Hashimoto Isabelle Augenstein Eduard Hovy Hannah Rashkin! Will announce a Best Paper Award ($500) and a Outstanding Paper

To CoT or not to CoT?🤔 300+ experiments with 14 LLMs & systematic meta-analysis of 100+ recent papers 🤯Direct answering is as good as CoT except for math and symbolic reasoning 🤯You don’t need CoT for 95% of MMLU! CoT mainly helps LLMs track and execute symbolic computation

CDS Faculty Fellow opening: Seeking interdisciplinary faculty fellows in ML, cognitive science, theory, responsible AI, natural sciences, social sciences, NLP & healthcare. 2-year position, competitive package. Apply by Nov 25 for Sep 2025 start. Info: apply.interfolio.com/153414

We studied retrieval diversity on subjective questions with different types of corpus (Wikipedia, web snapshot, search results)! This project made me think a lot about the future of retrieval system evaluations.

It was fun exploring augmenting in-context examples to retrieval (text embedding) models with Atula Tejaswi Yoonsang Lee Sujay Sanghavi! It doesn't work as magically with LLMs out-of-the-box, but in-context examples can help after fine-tuning.

Check out our new paper on KV compression for long text generation. Key insight: small KV cache needs to be refreshed occasionally!

When using LLM-as-a-judge, practitioners often use greedy decoding to get the most likely judgment. But we found that deriving a score from the judgment distribution (like taking the mean) consistently outperforms greedy decoding. Check out Victor Wang's thorough study!

Excited to announce our 2025 keynote speakers: Shirley Ho, Nicholas Carlini, Luke Zettlemoyer, and Tom Griffiths!

Can we generate speech that aligns with abstract, rich style tags (e.g., confused, authoritative)? Anuj's new work makes a step towards it through careful data augmentation!

We are receiving repeating questions about the double submission policy in relation to the abstract deadline. Our FAQ addresses this, and spells out what will be included in any double submission checks we do with other venues. Let us know if there are more Qs

Would LLMs think "Houston, Austin, Dallas" is sampled from "cities in Texas" rather than "cities in the US"? I really enjoyed our work exploring reasoning of LLMs about these suspicious coincidences!

Please check out Michael's #ICLR2025 poster on training LLMs to ask clarifying questions. LLMs are eager to answer immediately, even when the input is ambiguous. We simulate future turns and then assign rewards based on it, teaching LLMs to see a value in asking clarifying

I will be presenting this work at NAACL 2025! More specifically at 5pm on Thursday (May 1st), at Ruidoso. (Session IIS 1) Looking forward to catching up with old friends and meeting new people! Let’s chat!

Interested in how alignment changes the response distribution defined by LLMs? Come check out my poster at 2 PM at #NAACL2025 x.com/thomlake/statu…