Eliciting Latent Predictions from Transformers with the Tuned Lens Analyzes transformers from the perspective of iterative inference, seeking to understand how model predictions are refined layer by layer. repo: github.com/AlignmentResea… abs: arxiv.org/abs/2303.08112

Ever wonder how a language model decides what to say next? Our method, the tuned lens (arxiv.org/abs/2303.08112), can trace an LM’s prediction as it develops from one layer to the next. It's more reliable and applies to more models than prior state-of-the-art. 🧵

Timaeus is a new research organization, dedicated to making fundamental breakthroughs in technical AI alignment using deep ideas from mathematics and the sciences. Led by Jesse Hoogland Consistently Candid Alex Stan van Wingerden and myself. lesswrong.com/posts/nN7bHuHZ… [1/n]

1/8 How do transformers learn? In our new work, we find that transformers develop in-context learning in discrete stages that can be automatically discovered. 🧵 arxiv.org/abs/2402.02364 Joint work w/ george, Matthew Farrugia-Roberts, Liam Carroll, Susan Wei, Daniel Murfet

1/ How do attention heads form? With our new approach, we show that attention heads have distinct developmental signatures. These signatures reveal how heads develop distinct functional roles specialized to different subsets of data. In the process, we discover a new circuit.