I've always been fascinated by learning-to-learn. Now I've had the privilege of following along on Johannes Oswald journey to show that Transformers certainly do this in some settings, and likely do this in the wild! Quite extraordinary emergent behaviour!
arxiv.org/abs/2212.07677