New blog post where I discuss what makes an language model evaluation successful, and the "seven sins" that make hinder an eval from gaining traction in the community: jasonwei.net/blog/evals Had fun presenting this at Stanford's NLP Seminar yesterday!

Although the bitter lesson suggests that end-to-end eventually wins, this talk observes that at a given level of (compute, data, algorithm, architecture), there exists an optimal structure to add to make things work at all. But we often forget to remove these when more end-to-end

I think plot from Hyung Won Chung and Jason Wei if understood is worth more than any other education in AI

Similar analogies but in soccer: - Goalie: Hardware folks - Defender: Infra & data engineers - Midfielder: All-around SWE and ML engineers - Offense: General AI researchers - Striker: Star AI researcher training the biggest models and doing yolo runs - Coach: people managers

One of the most exciting opportunities on our team, apply!

There is no better exercise in clear communication than writing data annotation guidelines

Inspiring words from a young OpenAI engineer: “Why have I done well so far? I don’t think I’m smarter or more experienced than other people. But my competitive advantage is that I am willing to sit down and fully debug and completely understand code. I am willing to stay up late