We will keep improving our work. And we are wandering around at #NeurIPS2023.

Love how the LLM360 team share their Weights & Biases workspaces publicly in the Metrics section for both Amber and Crystal Coder 😍 44 loss and eval charts logged during training, all publicly browsable wandb.ai/llm360/Amber

There's too much happening right now, so here's just a bunch of links GPT-4 + Medprompt -> SOTA MMLU microsoft.com/en-us/research… Mixtral 8x7B @ MLX nice and clean github.com/ml-explore/mlx… Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

In coding tasks, CrystalCoder approaches StarCoder-base in accuracy. In language, CrystalCoder is comparable to Llama and MPT-7B. While previously #LLM builders had to choose between coding or language, CrystalCoder is optimal for both of these tasks. Check it out here:

Really?

CrystalChat is currently the best model in LLM360, on both English and Code. With our special multiphase training, CrystalChat demonstrates surprising ability early during the training: see the training trajectory here: wandb.ai/llm360/Crystal…. A paper is coming soon!

Wondering if we are ready to try on the ChatbotArena, lmsys.org, Hao Zhang, Willie Neiswanger, Aurick Qiao

It’s year 2024, and n-gram LMs are making a comeback!! We develop infini-gram, an engine that efficiently processes n-gram queries with unbounded n and trillion-token corpora. It takes merely 20 milliseconds to count the frequency of an arbitrarily long n-gram in RedPajama (1.4T

Huge congratulations to the Ai2 team! The OLMo series is a substantial contribution to the OSS LLM community with: - open training datasets - 500 checkpoints - analysis and evaluations Couldn’t be happier to see more projects with similar goals (cc EleutherAI)

Open access goes beyond the open of model weights; it also encompasses the release of comprehensive training recipes, from ELMo to OLMo🚀 Big shoutout to Ai2 , along with my respect to BigCode EleutherAI LLM360 for their commitment to FULLY transparent AI models.

Congratulations to friends and colleagues Shibo Hao, Yi Gu Zhiting Hu on the new project. And Maitrix is a cool name!

Super proud to have played a small part in this large model! Arctic is a 480B parameter MoE with 128 experts, with weights released under Apache-2.0. We are also publishing a series of blog posts that that details and demystifies large MoE training, check it out!

A fundamental skill of human developers is to mentally simulate and reason about code execution in natural language. Can we teach LLMs this skill? Excited to share our recent work with Ansong Ni Miltos Allamanis 🇪🇺 Arman Cohan Yinlin Deng Kensen Shi Charles Sutton @ ✈️ ICML 2024 🥐 at Google DeepMind!

🔥Introducing Pandora 🌏 🪐 a World Model that generates videos of world states with real-time language control 🎥🕹️ Simulate the world across domains in an _interactive_ way! check out more world-model.ai

A World Model that is real-time steerable with language command, and able to real-time reason at concept level in visual space. Time to move past LLM that lives in lingual world and enter the physical and sensory world! Zhiting Hu, Yann LeCun, MBZUAI

Proud to see K2 finally out! Months of hard work of an amazing team and GPUs🖥️. Being transparent and matching Llama2-70B's performance, we believe K2 can push the frontier of AI in the OSS way. Stay tuned for more updates, or explore on our website llm360.ai now.

Tempting, calling llm360 team mates now.

Open source is well defined in software development, which features collaboration, peer production. The open weights ~= .exe analogy is neat: weights are like code, but compiled ones that require significant effort to reverse-engineer, not the right level for peer production.

Very excited about this! DCLM already led to a great training set for language models, and there is (much) more to understand + more room for improvement here.

I am starting a new company: