📢 Last Call for Applications! Apply by May 31 to join one of our three in-person events this summer: 📆 Summer Institute on Law and AI: July 11-15, Washington, DC-Area 📆 Workshop on Law-Following AI: August 6-8, Cambridge University, UK 📆 Cambridge Forum on Law and AI: August

As AI agents near real-world use, how do we know what they can actually do? Reliable benchmarks are critical but agentic benchmarks are broken! Example: WebArena marks "45+8 minutes" on a duration calculation task as correct (real answer: "63 minutes"). Other benchmarks

🚨 AI Evals Crisis: Officially kicking off the Eval Science Workstream 🚨 We’re building a shared scientific foundation for evaluating AI systems, one that’s rigorous, open, and grounded in real-world & cross-disciplinary best practices👇 (1/2) evalevalai.com/research/2025/…

🚀Come join my team at RAND! We’re looking for research leads, researchers, & project managers for our compute, US AI policy, Europe, & talent management teams. All teams have urgent, important work to do & broad options for the future. Some roles close July 27⏰

Selling H20 (and potentially Blackwell?) chips to China gives up valuable leverage. Lennart Heim and I argue there's a smarter approach: let China access these chips remotely via the cloud. 1/

We wrote a paper last year about all the ways industry orgs could influence policy tl;dr: unsurprisingly, there's lots of places you could spend money to influence policy, and industry is massively outspending civil society orgs on AI arxiv.org/abs/2410.13042

Excited to be an Asterisk fellow and to be doing more public writing! And very impressed with the rest of this cohort ^^

Very excited that this systematic analysis is out! We found a bunch of failure modes, as well as interesting and surprising behaviours. Theres a lot more insight we can get from looking carefully at how models are solving evaluation tasks!