Blows my mind that model souping Just Works™️ Same model, same data, train 3-5 times with different seeds, 1-2 extra points on MMLU, Hellaswag, ARC, GSM8k, etc

Has anyone done a load test using vLLM with H200 vs. H100 (say a standard model, like llama-3.1-70B)?

Intelligence too cheap to meter? No, the price will stabilize or maybe increase once the party is over.

Just ran 80,000 simulations using LLMs and now I have a $80k bill with openai. Can the winning candidate please bail me out?

All right so far and that's scary.

Big news if true and it's true, so big news!

There is a subtle distinction between RL in RLHF and RL in domains with a clear reward signal that captures what we want like winning in games, correctness in math With a clear reward, RL is quite effective and can lead to novel sequences of actions (e.g. move 37). But, ... 👇

AI monoliths vs Unix Philosophy: The case for small specialized models. The current thinking in AI is that AGI is coming, and that one gigantic model will be able to reason and solve business problems ranging from customer support to product development. Currently, agents are

Frontier models get a single-digit performance on FrontierMath, a new benchmark! What's interesting is that o1-preview lags behind Gemini 1.5 Pro, after having spent all those reasoning tokens.

If you are not using structured outputs, you are doing yourself a disservice.

This paper, "Scaling Laws for Precision" [1], is pretty cool and has some important insights. For example, they say "the more data seen during pretraining, the more sensitive the model becomes to quantization at inference-time" [2], which has real implications for data curation,

"Data curation and synthetic data are very fruitful enterprises. We’ve only begun to address data about past model interactions and comprehensive user signals. The things that people refer to as data-flywheels, data engines, or closing the loop are very promising." Thank you🫡

> llm scaling hits a wall* > people take smaller models and specialize them (🙋‍♂️) > apple wins > apple swallows nvidia *not entirely true, but more on this later.