🔥The 9th Round of Easy Loan, Earn $40 Reward is in progress❗️ ⏰ Promotion Period: January 15th - Feburary 15th, 2025 👉 Register now and check more details at gate.io/campaigns/358

When reading AI benchmarks, aside from the fact that many of the AIs are (accidentally or on purpose) trained on the test set, many tests are just bad. MMLU likely maxes out at 90% or so because so many of the questions in it are just wrong. It is also uncalibrated in difficulty.

Hell of a Voight-Kampff answer.

I have taught an entrepreneurship class for 15 years. I just had an online Q&A with a group of entrepreneurs taking a similar class. As an experiment, I put their questions into o3. The answers were all very good. (The examples here are not from the students, but are typical)

As happens to every AI term once it becomes used in marketing (see also "agents" and "vibe coding"), I regret to inform you that no one agrees on a common definition of "digital twin," let alone what a digital twin actually does.

A weird thing about LLMs is that they just happen to do many things but almost all uses are undocumented. For example, GPT-4o is very good at helping farmers identify swine diseases. There is a lot of value in experts exploring & benchmarking how good LLMs are at various tasks.