What Is FlashAttention?

FlashAttention Explainer

FlashAttention is an algorithm for computing exact attention more efficiently on GPUs. It does not approximate the attention formula. Instead, it reorganizes the work to reduce reads and writes between high-bandwidth memory and much faster on-chip memory.

The algorithm divides queries, keys, and values into tiles. It processes blocks in a carefully ordered way and maintains running softmax statistics, avoiding materialization of the complete attention matrix.

Less memory traffic can make training and inference faster while lowering peak memory, especially for long sequences. Later versions improve parallelism, work partitioning, and support for newer hardware.

Speedups depend on sequence length, head dimension, precision, GPU, and software integration. FlashAttention does not solve every memory cost in a model, and it still has quadratic arithmetic. Benchmarks should use end-to-end workloads rather than kernel timing alone.

FlashAttention computes exact attention in memory-efficient tiles, reducing costly movement between fast and slow GPU memory.

Key terms include FlashAttention, exact attention, tiling, GPU, high-bandwidth memory, on-chip memory, softmax, memory traffic and kernel.

Small blocks fit in fast on-chip memory and are reused before being written back.

Running softmax statistics avoid storing the full attention matrix.

FlashAttention reduces memory movement rather than removing every pairwise calculation.

Practical speedups depend on hardware, shape, precision, and the rest of the model.

Explore more "Explainers"

Discover additional explainers across politics, science, business, technology, and other fields. Each explainer breaks down a complex idea into clear, everyday language—helping you better understand how major concepts, systems, and debates shape the world around us.