Really impressive! I’m glad we’re on the verge of a good surfing simulation…. (with eye tracking input soon? :))

We need a version of the Needle for model releases: - Day 1 benchmarks (SWE-Bench Verified, MMLU Pro, HLE) - the model predictably looks good; needle points to “good vibes”. - Day 2 benchmarks (Pelicans, EQBench…and eval bizarro world) - the model underperforms: the needle

Give me a good eval, and I shall move the world. - Archimedes (apocryphal)

Most takes on RL environments are bad. 1. There are hardly any high-quality RL environments and evals available. Most agentic environments and evals are flawed when you look at the details. It’s a crisis: and no one is talking about it because they’re being hoodwinked by labs

Never scale before you learn.

This was an LLM wars subtweet, and is both right and wrong in different ways. It’s wrong in the sense that you learn things at scale that you wouldn’t learn otherwise - so, on the contrary, scaling allows you to learn. You don’t want to overoptimise for lessons at smaller scales

Quick hiring call. We’re looking for full stack engineers to join our growing team at General Reasoning. We have more work than hands at the moment - a nice problem to have! - and are working with clients on some groundbreaking projects (the most excited I’ve been since my early LLM

Last week we found an issue with SWE-Bench, allowing agents to cheat by looking at future commits. Instead of celebrating the SWE-Bench Devs for quickly fixing the issue and being transparent, the HN crowd is dunking on them and drawing wildly inaccurate conclusions about