Great paper, arguing emergent abilities are only a function of pre training loss and not model/dataset size.

ie, if you (inefficiently) overtrain a small model to the loss of GPT4, you'd get all the abilities of GPT4.

arxiv.org/abs/2403.15796

demos n chill 8 thread

Dave garmin dashboard “measuring my brain juice”

Fun thought experiment: what if the input into Sora wasn't text, but the motion sensor data of a robot.

It turns its head, and the scene rotates. It lifts its arm, and a hand comes into view, etc. Doesn't need eyes.

Phi-3 'paper' TLDR

ORPO is the shampoo & conditioner 2 in 1 of RLHF

we've bundled too far

If the outputs are the same, but the means are different, Yann would be so much happier.

Too bad no one else would care.

I was so worried the big AI labs were no longer publishing their research and I'd be left behind.

But it turns out it's all still train big models on lots of data.

Technology is making us less conscious, but consciousness overall is increasing.

An interesting AI math question: can you generate text with higher entropy than human text with an LLM? I'm looking at you 'Backdoors of Claude' people.

If so, how can a compression machine also be a decompression machine?

When I read something that changes my mind, I find it hard to believe that this was caused by a change in the strengths of my neurons. Am I wrong?

This paper is an implementation of self-awareness masquerading as 'making quadratic attention more efficient'.

I'm not fine-tuning! I'm reality constructing, belief propagating, personality incepting.

what happens when the means of production becomes the labor as well?

I was pretty happy with my next move for black. Desperate times.

from me as you

AI-generated sad girl with piano performs the text of the MIT License

300,000 years ago System 2 came out from System 1. But suddenly, a few years ago and to everyone's shock, System 1 came out of System 2! Now, there's a rush to build System 2 again. And then, sometime in the future, it will build a new System 1, on some distant planet, probably.

Viscerally feeling that making a clean dataset for training AI is itself an AI problem. So many edge cases!

i'd really like to never see this ever again