How do Transformers really do in-context linear regression? 1 TF layer = 3 steps of a second-order method! Can’t be GD, which converges exponentially more slowly. Meanwhile, LSTMs are more like online GD; they don’t learn second-order optimization (likely due to limited memory).