I assumed this was the case for a while, that neurons being 'purposefully' superimposed with unrelated concepts as a result of n tokens >> n params training regime is the fundamental tradeoff of overtokened NNs/mechanism of outsized perf, but curious to hear other opinions.

I think there's an explanation for Keller Jordan's Muon performance from a mechint perspective, specifically, than the orthonormalized gradient updates reduce superposition/feature interference. If anyone's interested I can post more experiment details