Happy to share Linguini🍝, a benchmark to evaluate linguistic reasoning in LLMs without relying on prior language-specific knowledge. We show the task is still hard for SOTA models, achieving below 25% accuracy. 📄: arxiv.org/pdf/2409.12126

Last week we released the first paper of my PhD, "MEXMA: Token-level objectives improve sentence representations". We present a novel cross-lingual sentence encoder (CLSE) trained with both token and sentence-level objectives. Paper: arxiv.org/abs/2409.12737 1/n

Happy to share our team's work on Large Concept Models (LCMs), a new approach for language modeling that goes beyond standard token-based LLMs by operating in a multilingual and multimodal embedding space. Check out the full paper! 📄: ai.meta.com/research/publi…

Happy to see that Linguini, our benchmark for language-agnostic linguistic reasoning, has been included in DeepMind’s BIG-Bench Extra Hard (BBEH). Linguini remains challenging for reasoning models, being one of only two (hard) tasks where o3-mini doesn't show massive gains.

If you are attending ACL2025 join our oral presentation! Happening at 15:00 in room 1.86 🙂