Great to see MetaCLIP algorithm (arxiv.org/abs/2309.16671) desaturate SSL training distribution as SSL 2.0. What’s next in SSL or pre-training? From our data research perspective, it’s likely about how to automatically desaturate a training distribution.

Congrats Rob Fergus ! Big win for FAIR

A new start of FAIR and excited to be part of it.

Heading to #ICML2025 (first time). Excited to meet need friends and old friends and chat about foundational data research and co-design with training (MetaCLIP), SelfCite arxiv.org/abs/2502.09604 with Yung-Sung Chuang and LongVU arxiv.org/abs/2410.17434 with XIAOQIAN SHEN .

Thanks for the invited talk and happy to share our industrial insights on “scaling data alignment” from Meta CLIP (its wide adoption and what’s next) in the DataWorld workshop #ICML2025 . happy to chat offline about data research.

If you want a job at Meta, get hired by OpenAI.

Phase 1 of Physics of Language Models code release ✅our Part 3.1 + 4.1 = all you need to pretrain strong 8B base model in 42k GPU-hours ✅Canon layers = strong, scalable gains ✅Real open-source (data/train/weights) ✅Apache 2.0 license (commercial ok!) 🔗github.com/facebookresear…

Need to upgrade the CLAP?

Great to see an intern project grows into a big project that is landing many impacts. Thanks for the hard work throughout the way.

will continue to contribute to open research eco-system, while we can

thx, there’s definitely some latent structure among the worldwide data that English only data don’t have.

As LLM research transitions into large-scale production and intense competition, momentum in areas less directly related to LLMs (like CLIP) has slowed and de-focused. We hope these fields can endure and prove essential for long-term scientific progress.

Genie 3 by Google DeepMind looks impressive. Extending Sora/Veo-style text-to-video generation with multi-round 'camera prompts' is an exciting direction. I believe the action space in world models goes far beyond human interaction through camera prompts—it should encompass much

"We don't just passively perceive the world; we actively generate it. The real world drives our perceptions, but the brain is always making best guesses. Perception is a controlled hallucination, constrained by sensory signals from the outside world" — Anil Seth

Truly appreciate the authors of Molmo Molmo (from Ai2 and University of Washington) for promoting open research and adopting MetaCLIP. There are many forms of openness today—such as open APIs, open weights, and open-source for reproducibility etc. I view MetaCLIP and Molmo's research

why many epochs? i thought 1 epoch training gets a better next batch for generalization.