FlashAttention is widely used to accelerate Transformers, already making attention 4-8x faster, but has yet to take advantage of modern GPUs. We’re releasing FlashAttention-3: 1.5-2x faster on FP16, up to 740 TFLOPS on H100 (75% util), and FP8 gets close to 1.2 PFLOPS! 1/

Super excited to be part of the FlashAttention3 crew. Please check out our results!

We will be reopening I-80 EB only to vehicles single file at 9 pm due to the I-80 mudslide at Baxter. WB lanes remain closed.