Why are MRIs so damn heavy? They weigh over 10,000 pounds, or about 3 cars' worth. MRI design knowledge is hidden inside large companies, so I tried to work it out from first principles. 🧵 (1/12)

this is a surprisingly creative (very) short story to have been written by an LM!

very interesting talk on the future of mathematics given language models: youtu.be/vYCT7cw0ycw?si… one thing this made me realize is that the next decade or so of math is going to be incredibly fun, as the reasoning models crank through all sorts of new and interesting questions

I wonder how much compute it would take to reproduce the entire published set of deep learning knowledge. if you were efficient about it.

I wonder if companies training creative models will be more durable than companies training models to perform well-defined tasks. Because it's easier for competitors to hill-climb on the evals for the tasks, but training a creative model requires the founders to have good taste,

Has anyone tried “sub-token attention”? Artificially increase the sequence length by including K copies of each token next to each other (say, each linearly projected by a different map), and let the different copies attend to each other. True self-attention :P (And then at the

One part of SF Compute we haven’t talked about very much yet is that post-AGI (presumably soon), the models will want to train more models. (Really, people will ask the first models to train more models, or perhaps to solve tasks that would benefit from, say, some custom RL).

very excited the weights of o1 finally arrived

Very excited for this new cluster. Big enough to train R1, but it's running our combinatorial auction so the prices should be rational

Feels like a good time to start a computer control startup. The methods are generally known (RL on top of base models), and it probably doesn't require that much compute, just thoughtful environment design. I would probably start with a text-only representation of websites.

I hope that somebody starts a company to make an AI-native smartwatch. It feels to me like the ideal form factor for most of what I want a language model to do.

This one seems like a good idea to me, increasingly I think datasets and RL environments are the limiting factor:

The new google image model is quite good except for the fact that it doesn't like to draw physicists:

I wonder what you would get if you trained something Cycle-GAN-like between images and music. Probably possible today with the quality of generative models we have!