We don't talk about t̶e̶s̶t̶i̶n̶g̶ eval coverage nearly enough. The entire point of LLMs is to have systems that are super robust to random user interactions and produce useful outputs despite that randomness.

OK - but what does sufficient eval coverage look like? good eval coverage = converging benchmark metrics -- each net new test case added to the eval benchmark yields diminishing returns.

Some of best consumer AI products out there started off as internal enablement tools. It’s the best choice if you decide pre-prod evals aren’t going to cut it and you don’t want to test in *real* prod (user-facing).

IMO benchmarking on real user data should be the golden standard for LLM providers releasing new models, especially top 5 ones. It's not hard to build such a benchmark (smaller model trainers are doing it!) so avoiding it for too long in the favor of off the shelf ones is odd.

Congrats Freddie Vargus Sara Hooker and everyone else from Cohere For AI’s Aya project! Super well deserved.

Human-aligned evals are key. The best AI products have domain SMEs involved in the eval creation process.

This and this. AI systems *and* AI evals get better when they integrate a nugget of human feedback.

"Do I need a reference dataset for evals?" Yes and no. You need to start w/ a set of test inputs. You need to have a definition of good for your system. Reference-based evals need a source-of-truth answer. Reference-free ones (e.g. w/ judge LLM) don't but must be human-aligned.

It's not just "regulated industries" who need to care about AI evals, same as it's not just finance or healthcare developers who implement unit testing. Testing before shipping is a good developer practice, whether it's for AI or not.

LLM work so far has been largely solo developers pushing independently. It's all about to get A LOT more collaborative. This small feature feels like it could be a big leap in using LLMs for collaborative work — as every Artifact can be shared, used by others, and “remixed”.

🔥 We’re excited to share that Quotient AI has been selected for the Hot AI List by The AI Furnace 🧨🔥 We look forward to what’s next as we continue shaping the future of AI, together. Onwards 🚀