📢📢 We are releasing TxT360: a globally deduplicated dataset for LLM pretraining 🌐 99 Common Crawls 📘 14 Curated Sources 👨‍🍳 recipe to easily adjust data weighting and train the most performant models Dataset: huggingface.co/datasets/LLM36… Blog: huggingface.co/spaces/LLM360/…

Someone confronted on the spot, and they said “ Maybe there is one, maybe they are common, who knows what. I hope it was an outlier.” Even this explanation is full of implicit racial bias. See the full conv: dropbox.com/scl/fi/2dtji0z…

It is just so sad that the #NeurIPS2024 main conference ended with such a racist remark by a faculty when talking about ethics. How ironic! I also want to commend the Chinese student who spoke up right on spot. She was respectful, decent, and courageous. Her response was

Today’s event was a glaring display of ignorance and deep-seated bias, proving that academic achievement doesn’t equate to decency or awareness. I feel for those who have to tolerate and work with her every day. Massachusetts Institute of Technology (MIT) and NeurIPS Conference, this is unacceptable—you can and must do

I saw a slide circulating on social media last night while working on a deadline. I didn’t comment immediately because I wanted to understand the full context before speaking. After learning more, I feel compelled to address what I witnessed during an invited talk at NeurIPS 2024

At #NeurIPS2024, the keynote speaker perpetuated explicit racist stereotypes against Chinese students. Generalizations against a community subject to discrimination, even as an “example”, further provokes discrimination. Below, video of an audience member’s perfect response.

I’m not at NeurIPs this year but I have seen the deeply offensive and racist slide shown by a keynote (not reposting here). It’s completely unacceptable, and I stand with my Chinese colleagues.

Just can't believe this happened at NeurIPS and, ironically, from an invited keynote speaker talking about ethics! Removing racial bias from humans is so much harder than removing it from LLMs. So proud of the Chinese student who speaks up on spot, pointing out the racist

I have three Ph.D. student openings in my research group at Rutgers Computer Science Department starting in Fall 2025. If you are interested in working with me on efficient algorithms and systems for LLMs, foundation models, and AI4Science, please apply at: grad.rutgers.edu/academics/prog… The deadline is

The LLM360 team is hiring! 🚀 We welcome researchers and ML engineers to join our team to push the frontier of AI research. You'll be deeply involved and gain firsthand experience in large model training. Join us in our mission of making AGI open to all. For more information,

I spent the weekend reading some recent great math+reasoning papers: 1. AceMath (arxiv.org/abs/2412.15084) 2. rStar-Math (arxiv.org/pdf/2501.04519) 3. PRIME (arxiv.org/abs/2412.01981) Here are some of my naive thoughts! It could be wrong. All of these papers are showing possible

We are releasing our code base for TxT360, a globally deduplicated dataset for LLM pretraining: 🌐 github.com/LLM360/TxT360 (You can access the full dataset here: huggingface.co/datasets/LLM36…) We are happy to see the open source community filled with great recent dataset releases,

LLM360 gets way less recognition relative to the quality of their totally open outputs in the last year+. They dropped a 60+ page technical report last week and I don't know if I saw anyone talking about it. Along with OLMo, it's the other up to date open-source LM.

Transformers vs Mixture of experts in LLMs visually explained:

Tired of complaining about *CL reviews on social media and conferences? Now’s your chance to make a real difference! Your feedback is invaluable in improving the process. Take a few minutes to share your thoughts—every response counts!

Strongly agree!

Proudly present MegaMath, the largest open-source math reasoning pretraining corpus—371B tokens of high-quality mathematical web, code, and synthetic data, designed to build the data foundation for next-generation math-proficient LLMs like o1 and R1. 🧵👇 #LLM #OpenSource

Great work from Fan Zhou and the MBZUAI data team!

📢📢 TxT360 has been updated to v1.1: 🌟 BestofWeb: high-quality doc set from the web ❓ QA: Large Scale Synthetic Q&A dataset 📖 Wiki_extended: extended wiki articles via links 🌍 Europarl Aligned: reformatted long aligned corpus huggingface.co/datasets/LLM36… #AIResearch

Efficient training of neural networks is difficult. Our second Connectionism post introduces Modular Manifolds, a theoretical step toward more stable and performant training by co-designing neural net optimizers with manifold constraints on weight matrices.