New paper! 'Universal Neurons in GPT2 Language Models'
How many neurons are independently meaningful?
How many neurons reappear across models with different random inits?
Do these neurons specialize into specific functional roles or form feature families?
Answers below 🧵:

After computing maximum pairwise neuron correlations across 5 different models trained from different random inits we find that (a) only 1-5% are 'universal'; (b) High/low correlation is one model implies high/low correlation in all models; (c) neurons depth specialize

What properties do these universal neurons have? They seem to consistently be high norm, sparsely activating, with bimodal right tails. In other words, what we would expect of monosemantic neurons!

When we zoom in, many neurons do have relatively clear interpretations! Using several hundred automated tests, we taxonimize the neurons into families, eg: unigrams, alphabet, previous token, position, syntax, and semantic neurons

We also observe many neuron functional roles. For instance (a) prediction, (b) suppression, (c) partition neurons which make coherent predictions about what the next token is (not). Suppression neurons reliably follow prediction neurons (bottom)

We found a very special pair of high norm neurons (which exist in all model inits) which do not compose with the unembed. Instead of changing the probability of any individual tokens, they change the entropy of the entire distribution by changing the scale!

Attention heads can be effectively 'turned off' by attending to BOS token. We find neurons which control the amount heads attend to BOS, effectively turning individual heads on or off.

There were lots of mysteries we didn't fully understand. One fairly striking one was the relationship between activation frequency and the cosine similarity between input and output weights!

See the full paper for all the details
Paper: arxiv.org/abs/2401.12181
Code: github.com/wesg52/univers…