🎉 Excited to share “Generalizing from Short to Long: Effective Data Synthesis for Long-Context Instruction Tuning” 📄 (arxiv.org/pdf/2502.15592) We propose "context synthesis": instead of generating instructions from long texts, we synthesize contexts for instructions—drawing

Q-Filters: Leveraging QK Geometry for Efficient KV Cache Compression

We find a single biased direction encodes a KV Cache selection mechanism in Self-Attention -- Key vector with a strong component in this direction results in this Key-Value pair being ignored by Query🚀🚀🚀

*Q-Filters: Leveraging QK Geometry for KV Cache Compression* by Nathan Godey Alessio Devoto Yu Zhao Pasquale Minervini Benoît Sagot We find directions in the KV cache geometry allowing us to compress the cache significantly with little degradation in performance. arxiv.org/abs/2503.02812

I'm looking for 2 emergency reviewers for ACL 2025 in the Language Modeling and Efficient methods for NLP tracks Please reach out in my DMs if you are interested and can do a review within 24 hours 😬

ModernBERT or DeBERTaV3? What's driving performance: architecture or data? To find out we pretrained ModernBERT on the same dataset as CamemBERTaV2 (a DeBERTaV3 model) to isolate architecture effects. Here are our findings:

Excited to introduce 𝗕𝗶𝗼𝗺𝗲𝗱-𝗘𝗻𝗿𝗶𝗰𝗵𝗲𝗱 🎉, a new annotated biomedical dataset designed to tackle the scarcity of clinical data for NLP research! 133M paragraphs from PMC-OA annotated for type, domain, and educational quality and publicly available on Hugging Face👇🧵

We produced FineWeb-Edu style annotations for biomedical data and showed that it helps for continued pre-training and lets us target domains to improve on! Work led by the amazing rian and supervised by Villemonte de la Clergerie Éric 🌟 Check out the thread and paper below 👇🏼

🏆 Our NVIDIA KV Cache Compression Leaderboard is now live! Compare state-of-the-art compression methods side-by-side with KVPress. See which techniques are leading in efficiency and performance. 🥇 huggingface.co/spaces/nvidia/…

.Cornell University is recruiting for multiple postdoctoral positions in AI as part of two programs: Empire AI Fellows and Foundational AI Fellows. Positions are available in NYC and Ithaca. Deadline for full consideration is Nov 20, 2025! academicjobsonline.org/ajo/jobs/30971