I just published a blog on PyTorch Tensors at a low level 3 Key Learnings: 💿 Strided Tensors store data contiguously in memory, using metadata (shape + stride) to describe access 🤖 Autograd builds dynamic computation graphs to automatically compute gradients—perfect for rapid

Luminal can discover flash attention entirely automatically. We've been working towards this north star in our search compiler. Check out the prototype demo below ↓

Since we’ve got a lot of new people following Luminal's progress, I figure we should go over where we are and where we’re going ↓

Link here: medium.com/data-science-c…

i can't square this with the other hazy post about how tensor cores make up 95% of the flops on a GPU. wouldn't it be massively beneficial? or is it because matvecs are so bandwidth constrained compared to matmuls? if someone from hazy follows me or knows someone from hazy lmk!