We are releasing a family of dense and MoE language models with up to 13B and 1.1T parameters. We find that MoEs are more efficient, but the gap narrows at scale and varies greatly across domains and tasks.

Paper: arxiv.org/abs/2112.10684

Models & code: github.com/pytorch/fairse…

Models and code available in fairseq: github.com/pytorch/fairse…

Mixture of experts training in fairseq is now 40% faster thanks to Microsoft's Tutel library!
Blog: microsoft.com/en-us/research…
Fairseq code: github.com/pytorch/fairse…
Tutel code: github.com/microsoft/tutel

We’re introducing GSLM, the first language model that breaks free completely of the dependence on text for training. This “textless NLP” approach learns to generate expressive speech using only raw audio recordings as input. Learn more and get the code:
ai.facebook.com/blog/textless-…

fairseq now supports CPU offloading and full parameter+optimizer state sharding via fairscale's FullyShardedDataParallel module. See our tutorial to train a 13B parameter LM on 1 GPU: fb.me/fairseq_fsdp

We just released 0.10.0, which is our last significant release before 1.0.0 when we will migrate to Hydra. Changelog: github.com/pytorch/fairse…

Facebook AI Research's sequence modeling library fairseq has made it's twitter debut. Please follow for latest updates.

Fairseq includes support for sequence to sequence learning for speech and audio recognition tasks, faster exploration and prototyping of new research ideas while offering a clear path to production. bit.ly/2WfP85X

roberta = torch.hub.load('pytorch/fairseq', 'roberta.large')

fairseq now supports the training of gated convolutional language models (arxiv.org/abs/1612.08083). It can train a Google Billion Word language model on 128 GPUs in less than a day.

FairSeq Toolkit - Major Update
- Distributed Training
- Transformer models (big Transformer on WMT Eng-German in < 5 hours on DGX-1)
- Fast Inference: translations @ 92 sent/sec for big Transformer
- Story Generation
Read more at Michael Auli's post: facebook.com/photo.php?fbid…

Fairseq, now in PyTorch!
The open-source convolutional sequence-to-sequence engine from FAIR is now available in... fb.me/1gCPauX6V