Why do you recommend binary (pass/fail) evaluations instead of 1-5 ratings (Likert scales) for applied evals? Links in reply

Should I build a custom annotation tool or use something off the shelf? P.S. This is a correction - I wasn't opinionated enough before. Links in reply

This will be a banger. Jim Blomo will discuss how OpenAI customers use evals w/ real case studies & results. Did you know OpenAI has eval tools (traces, scoring ,etc)? He will show that too. Recording made available to all that sign up. maven.com/p/d2dc30/how-o…

I believe this is the first talk of its kind - we get to hear from OpenAI on best practices for applied Evals with **real case studies**. We will also get a sneak peek of OpenAI's up and coming eval tools. maven.com/p/d2dc30/how-o… Will be recorded

Hamel Husain We need "start simple, look at your data, and don't get distracted by fancy tools before you understand what's really happening" billboards

Like 95% of people are shocked that people don't look at their data enough, and yet 95% of people don't look at their data enough. The math ain't mathing. That's b/c there is MUCH more depth & nuance than most realize. Sign up to learn what you need to do in practice and how.

One third of folks are cooked

💯”They taught me this whole process: test your AI with different types of questions, write down everything that goes wrong, then figure out patterns. Sounds boring, but here's the thing, I was missing so many errors I didn't even know existed…Instead of just crossing my fingers

Real friends share their prompts and AGENTS .md files with each other 😍

I want to give a huge thanks to Shreya Shankar and Hamel Husain for leading an amazing course on AI Evals! They've laid out in such simple yet intuitive steps how to improve AI apps w/ data-driven evidence. The 3 Gulphs has been a real game changer in my way of AI problem-solving!

I have been enjoying the "AI Evals For Engineers & PMs" course from Hamel Husain and Shreya Shankar. I knew when I signed up I was going to be told "look at your data!". No surprise there! But this course teaches you how to do that using a scientific process. If looking at your data

1000x ROI! Wow 🤩 🚀 And yes…totally agree effective evals means acting more like an ML engineer & doing structured error analysis👷

“I knew this stuff was important before the course, but I feel much better equipped to do it in practice now. And I'm learning so much knowledge that I can take back to my company to teach others.” Awesome! 😎

For 3 weeks I’ve had the privilege to take the AI Evals course from Hamel Husain & Shreya Shankar. My team has known we need to skill up on evals and we all read what we can but there is no comparison to lecture + course reader + homework + guest presentations + office hours