"Spectral autoregression" is a useful but overly-simplified anology of diffusion. IMO diffusion training is constructing countless feature/semantic neighborhoods, and inference is randomly walking along these neighborhoods. The guidance signal suggests the trajectory of the walk

Inference scaling law is not new. It's been adopted by diffusion models 😛

Updating matmul to einsum in all my code 😂

Haha interesting artifact of maximum likelihood decoding

This won't age well 🤷‍♂️

Excited to witness a time when AIs are (rightfully) more trusted than their human creators 😆

Nice discovery and maybe we can "scale up" the conclusion: maybe at the end of the day, every LLM, including the largest ones nowadays, is a "small model", in the sense that they have difficulty in generating certain challenging CoTs

Alignment really gets interesting these days. On the surface this might look like self-consciousness, however I'd hypothesize that the prompt makes R1 enter some kind of role-playing mode so it fakes its viewpoints.

Make America Make Again 😎

Wow these poll numbers blew up my mind

Good insight. RL agents go full simulation mode

That's why autoregressive models are a perfect candidate for causality modeling

Imagine in the future such news is fed to Grok for training. It would be deeply buried in the weights and hard to rectify

One of the best episodes of Dwarkesh's interviews. I like that Karpathy always offers thought-provoking insights whenever he pushes back on an idea or a question.

So we can inject random vectors to let the model meander like humans🤯

Such a bad take. 99％ bugs/exploits are discovered by programmers who intended to use them in legitimate ways.

RL training

A more practical "Einstein test" is to train an LLM on pre-bitcoin text and see if it can propose and implement bitcoin