🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

Did you know that O1/R1 do not have access to their own thoughts from earlier in the conversation? Is it a well-known fact? For example, if you ask R1 to play a guessing game, it will pick a word, but in the following turns, it will have no idea what it picked. 😱 The same

🌟Want to know more about our MTRAG benchmark? Check out the IBM blog highlighting our work! research.ibm.com/blog/conversat… IBM Research

(1/N) LLMs represent numbers on a helix? And use trigonometry to do addition? Answers below 🧵

👉Evaluating LLMs?👈 We’d love to hear from you. Which best describes what you do? (comment for other uses)

General intelligence but they can't read a simple table or figure that changing the column does impact the content. Massive table benchmark:

The leaderboard for our new benchmark in collaboration with Stanford’s HELM powered by Unitxt! Take a look top models really struggle! crfm.stanford.edu/helm/torr/late…

"Summarize this text" out ❌ "Provide a 50-word summary, explaining it to a 5-year-old" in ✅ The way we use LLMs has changed—user instructions are now longer, more nuanced, and packed with constraints. Interested in how LLMs keep up? 🤔 Check out WildIFEval, our new benchmark!

How do LLMs cope with multi-constraint instructions from real users? Not too well, it turns out... So lots of room for improvement! 🦾 Great internship work by Gili Lior 🌟

Care about LLM evaluation? 🤖 🤔 We bring you🕊️ DOVE a massive (250M!) collection of LLMs outputs On different prompts, domains, tokens, models... Join our community effort to expand it with YOUR model predictions & become a co-author!

🚨 Introducing LAQuer, accepted to #ACL2025 (main conf)! LAQuer provides more granular attribution for LLM generations: users can just highlight any output fact (top), and get attribution for that input snippet (bottom). This reduces the amount of text the user has to read by 2

Important reminder to the world