What is flash attention?
Flash attention is a way of computing a transformer's attention operation without ever writing the full attention matrix out to GPU memory. The math is identical to standard attention; what changes is how much data moves in and out of memory.
Standard attention computes a score between every pair of tokens, producing a matrix that grows with the square of sequence length. It writes that matrix to memory, reads it back for the softmax, and reads it again to combine it with the value vectors. Flash attention fuses these steps into one kernel and processes the sequence in tiles, keeping intermediate results in the GPU's fast on-chip memory and writing only the final output.
Why memory traffic, not compute, is the bottleneck
The obvious assumption is that attention is expensive because of the number of multiplications, and at long context lengths that's part of it. But on modern GPUs, moving data between main memory and the on-chip compute units is often slower than the arithmetic. Standard attention writes the full N-by-N score matrix to memory and reads it back several times, and that matrix grows fast: double the sequence length and it quadruples. Flash attention keeps those intermediate values in fast on-chip memory, which cuts memory traffic while the number of floating-point operations stays roughly the same.
What it changes in practice
Two things follow from lower memory traffic. First, attention's memory use scales linearly with sequence length instead of quadratically, so a model can handle a longer context window in the same memory. Second, attention spends less time waiting on memory, so that part of the forward pass runs faster, especially during prefill, when the whole prompt is processed at once and the attention matrix would otherwise be largest.
The GPUwerk benchmarks page shows prefill on the Spark at around 1,900 to 1,950 tokens per second, the compute-bound regime where a kernel like this helps. Decode is bounded by the memory bandwidth needed to read model weights, a separate bottleneck flash attention doesn't touch.
Why longer context needs it
Without a fused kernel, doubling context length roughly quadruples the memory the attention matrix needs. That wall kept very long context windows impractical until kernels like this became standard in inference engines. See handling long context requests for the serving side, and what is a context window for the concept itself.
Where it fits in a typical inference stack
Flash attention is a kernel inside inference engines such as vLLM and llama.cpp. In most setups you don't configure it separately; it's on by default when the hardware and model support it. Together with optimizations like KV caching, it determines how much context and how many concurrent requests a given amount of memory can serve. See the choosing an inference engine guide for how engines differ on this.