The reality of generative AI is that it often makes mistakes, just like humans! But why would AI startups allow that to cause hesitancy for users considering adoption? Why would you price based on usage when you could price based on successful usage? Mutable validates outcomes.

If there’s anything that needs to be closed source and secret and run only by a few trusted capable people, it’s good high quality evals. The closed nature is primarily to ensure training datasets don’t get contaminated. The “few” is to ensure we don’t have noise and low quality

I wonder if there's some sort of business to be made creating a regularly updated private benchmark.

👋