LLMs “intelligence” is hard to benchmark, as we don’t have good benchmarks for human performance at complex tasks.

Take theory-of-mind: several tests found GPT-4 beats humans, but another one finds a huge gap. Is it the testing structure? Prompting? Which is right? Hard to know.

I now believe that AI automated summaries of customer reviews are clearly always good and accurate.

We've seen music and voices, but now you can make AI generated sound effects from text. Another piece of the 'completely AI generated media' puzzle falls into place.

I gave the elevenlabs version a go, not perfect, but pretty neat. This one was first try.

More evidence that from learning styles to love languages to Meyers-Briggs to horoscopes, you can't look at a complex phenomena (relationships, education, personalities) and easily categorize people into a set of universal & distinct groups with non-overlapping characteristics.

Papers:
Strategy professors: papers.ssrn.com/sol3/papers.cf…
Kenyan founders: osf.io/preprints/osf/…
Consultants: papers.ssrn.com/sol3/papers.cf…

Growing evidence that GPT-4 class LLMs give good business advice if done carefully:
1) Getting AI advice boosts profits of high performing Kenyan entrepreneurs by 18%
2) Our paper shows that AI can help consulting work at a high level
3) AI and strategy professors agree on advice

It is worth being careful about the persona you are assigning the AI, it gives the LLM context, not magic powers. Papers show it often helps, but it requires testing.

For example 'you are a sophisticated prize-winning author' produces a purple-prosed parody of high-end writing.

Incidentally, the threshold for proposed regulation in many jurisdictions is 10^26 FLOPs. The charts show this will be a barrier that will be substantially passed by frontier models this year and most models in a couple years, if trends continue.

The computing power used to train frontier AI models (and all AI models) is increasing at over 4x a year.

Until we see a sign of slowdowns in investment and returns to compute (not yet, but which will happen eventually), this is the base trend to expect. epochai.org/blog/training-…

Useful set of warnings about LLMs in legal work - we have few benchmarks to know what they are good or bad at.

But I am not sure it will matter. From both published use cases (Moderna, etc.) & conversations, it seems actual adoption is happening quickly. arxiv.org/pdf/2402.01656

The most common LLM failures you see shared on Twitter are word games - AI is really bad at working with text positions ('give me 10 sentences that end with the word apple' 'give me 3 countries that end with a k'). This approach suggests that the problem may be solvable.