Very excited to present our paper "Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysis" at ICML Conference 2024! Come see our poster tomorrow, Wed. July 24th 1h30-3pm Paper: openreview.net/forum?id=hLuNV… Code: github.com/shoroi/align-n… Mila - Institut québécois d'IA #ICML2024

How do MoE transformers, like DeepSeek, behave under distribution shifts? Do their routers collapse? Can they still match full re-training performance? Excited to present “Continual Pre-training of MoEs: How robust is your router?”!🧵arxiv.org/abs/2503.05029 1/N

Mes remerciements les plus sincères à la Fondation Schulich, à M. Seymour Schulich et à l'Université de Montréal! #2017SLSquad