OpenDiloco update: I think that we hit the sweet spot with our last experiments. We managed to match the loss of the baseline with 200x less communication. The key was to trade the amount of inner step with more quantization on the pseudo gradient. We are preparing a

The hottest open-source decentralized model in town

Diloco was published the 14 Nov 2023, and on the 17 Nov, 3 days after, Ilya saw something. Coincidence? 🤔

Ilya seems excited about distributed training 👀 github.com/PrimeIntellect…

(((ل()(ل() 'yoav))))👾 Anna Rogers

Today, we are rolling out three experimental models: - A new smaller variant, Gemini 1.5 Flash-8B - A stronger Gemini 1.5 Pro model (better on coding & complex prompts) - A significantly improved Gemini 1.5 Flash model Try them on aistudio.google.com, details in 🧵

Chatbot Arena update⚡! The latest Gemini (Pro/Flash/Flash-9b) results are now live, with over 20K community votes! Highlights: - New Gemini-1.5-Flash (0827) makes a huge leap, climbing from #23 to #6 overall! - New Gemini-1.5-Pro (0827) shows strong gains in coding, math over

Recall when people were talking about trillions-of-params-OG-gpt4? This model was tbh insane. Now, on human preference thru lmsys, a small and ultra fast small Gemini-flash model would beat it. That's acceleration.

Two months back, we released a 9B RecurrentGemma model, one of the strongest SSM-based language models out there, trained on 2T tokens! I finally updated arXiv with some of our results: arxiv.org/abs/2404.07839 Link to weights and code for our models in thread!