📢 Today, we release #MMLUProX, which upgrades MMLU-Pro to 29 languages across 14 disciplines—11,829 reasoning-heavy Qs per language (≈342 k total). The toughest multilingual stress test for today’s LLMs! 🌐🧠 Heartfelt thanks to everyone who contributed.🤝

🚨 Announcing DeepTheorem: Revolutionizing LLM Mathematical Reasoning! 🚀 𝕋𝕃𝔻ℝ: - 🌟 Learning by exploration is the most important rationale that recent RL-zero training teaches us since self-exploration significantly boosts the utilization of LLM pre-training knowledge; -

I am pleased to announce that I will be joining Johns Hopkins University's Computer Science Department JHU Computer Science as an Assistant Professor in Fall 2025. I am grateful to my mentors for their unwavering support and to my exceptional PhD students for advancing our lab's vision.

I’m thrilled to share that I will be joining Johns Hopkins University’s Department of Computer Science (JHU Computer Science, Johns Hopkins Data Science and AI Institute) as an Assistant Professor this fall.

When eyes and memory clash, who wins? 👁️🧠 Introducing a comprehensive study on vision-knowledge conflicts in MLLMs, where visual input contradicts the model's internal commonsense knowledge—and the results might surprise you. #ACL2025NLP 📈 We developed an automated framework

Can MLLMs truly "see" safety risks in image-text combinations? 🌲🖼️ Introducing MMSafetyAwareness, the first comprehensive benchmark for multimodal safety awareness in MLLMs, featuring 1,500 image-prompt pairs across 29 safety scenarios to evaluate whether models correctly

🤔How well do LLMs adapt to different norms? 🧵We introduce STEER-BENCH, a benchmark for assessing steerability in LLMs. 📉 Human: 81% | Top LLM: ~65% 🚨 Norm alignment ≠ solved. 📄 Paper: arxiv.org/abs/2505.20645 Zihao He Taiwei Shi 🇺🇦 Kristina Lerman 🇺🇦

Congrats Minghao!!!

Let’s jailbreak step by step!

What if LLMs could learn your habits and preferences well enough (across any context!) to anticipate your needs? In a new paper, we present the General User Model (GUM): a model of you built from just your everyday computer use. 🧵

Do LLMs truly understand math proofs, or just guess? 🤔Our new study on #IneqMath dives deep into Olympiad-level inequality proofs & reveals a critical gap: LLMs are often good at finding answers, but struggle with rigorous, sound proofs. ➡️ ineqmath.github.io To tackle

Our new paper explores knowledge conflict in LLMs. It also issues a word of warning to those using LLMs as a Judge: the model can't help but inject its own knowledge into its decisions.

Interesting work!

We have a long way to go on visual reasoning. Our VisualPuzzles benchmark🧩shows similar findings, where the best models still can’t beat the bottom 5% of humans. 👉Check out our threads: x.com/yueqi_song/sta…

Happening now at East 1101!

Looking for a high-quality dataset? Come to west 414 NOW