🥇 I won the the tiny corp x AGI House SF hackathon solo (+support of George Hotz 🌑) Hacked an @apple Mac Mini distributed AI training cluster with thunderbolt-4 interconnect (40Gbps) for linear scaling Detailed post coming soonTM Exo Labs Emad

🤔 extracted the full ~5000 token claude3.5sonnet claude.ai system prompt: gist.github.com/1rgs/b31a1de86… this is a great template for function calling / tool use notes: artifacts: seem to be a fully in-context abstraction, model not finetuned for it allowed types:

Rejection sampling is commonly used to synthesize SOTA SFT data for math reasoning, which filters out the synthetic responses with incorrect answers. However, have you realized that this process may implicitly introduce bias towards easy samples? — Easy queries become more and

This is the crazy eval I've wanted to see for so long!!

I’ll be attending ICML Conference next week presenting my long context data engineering paper! Come and discuss long context architecture, data, inference efficiency, and I will reason with you why long context is much better than RAG arxiv.org/abs/2402.10171

✨New Preprint ✨ How are shifting norms on the web impacting AI? We find: 📉 A rapid decline in the consenting data commons (the web) ⚖️ Differing access to data by company, due to crawling restrictions (e.g.🔻26% OpenAI, 🔻13% Anthropic) ⛔️ Robots.txt preference protocols

Modern LLMs have large vocab sizes and long seq lengths which leads to an annoying peak in memory due to logit activations... .... so, I wasted some time last month writing a fused triton kernel to do nn.Linear + nn.CrossEntropyLoss without a memory peak ⬇️

A proper coffee at ICML Conference

Great to see DART-Math is featured in NuminaMath's report and outperforms DeepseekMath-RL by 5 points on OOD test, when our model is trained only on synthetic data from DeepseekMath-RL itself.

Albert is one of the most thoughtful researchers and one of the people in early days of AI4Math, particularly in early days of autoformalization when nobody (including myself 😅) understood the importance of it (if you still in question, check AlphaProof) So huge congrats on

Ai2 released OLMoE today. It's our best model to date. - 1.3B active, 6.9B total parameters, 64 experts per layer - Trained on 5T tokens from DCLM baseline + Dolma - New preview of Tulu 3 post training recipe - Fully open source - Actually SOTA for ~1B active params I'm most

🚀 Introducing MMMU-Pro: A more robust version of MMMU arxiv.org/pdf/2409.02813… After launching MMMU, we received valuable feedback from the community: 1️⃣ Some questions were answerable without even seeing the images. 2️⃣ Models didn’t always "know" the answer but found shortcuts

We’re presenting AlphaProteo: an AI system for designing novel proteins that bind more successfully to target molecules. 🧬 It could help scientists better understand how biological systems function, save time in research, advance drug design and more. 🧵 dpmd.ai/3XuMqbX

Looking back, the largest problem of my own phd journey is reading too many papers and writing too few codes 😮‍💨

Automating AI research is exciting! But can LLMs actually produce novel, expert-level research ideas? After a year-long study, we obtained the first statistically significant conclusion: LLM-generated ideas are more novel than ideas written by expert human researchers.

What do people use ChatGPT for? We built WildVis, an interactive tool to visualize the embeddings of million-scale chat datasets like WildChat. Work done with Wenting Zhao Jack Hessel Sean (Xiang) Ren Claire Cardie Yejin Choi 📝huggingface.co/papers/2409.03… 🔗wildvisualizer.com/embeddings/eng… 1/7