1. Introduction
Transformer attention computes
where , , and , is the sequence length, and is the head dimension. The factor scales the dot products before softmax. This formulation suggests a straightforward implementation: compute the score matrix , apply row-wise softmax to obtain , and compute .
The straightforward implementation is inefficient because both and contain elements. Each intermediate is written to HBM and read by a subsequent kernel. GPU tensor cores can compute the matrix products rapidly, but the intermediate traffic repeatedly crosses a much slower memory interface.
FlashAttention is based on one systems observation:
The kernel computes one score tile in on-chip SRAM, immediately incorporates it into a running softmax and output, and then discards it. This reordering computes the same attention function with substantially less memory traffic.
2. Background
2.1 GPU memory hierarchy
GPUs expose a large HBM capacity and a much smaller on-chip storage hierarchy composed of registers and shared memory. HBM holds model state and activations but has much lower bandwidth relative to the arithmetic rate of tensor cores. Shared memory is fast enough to feed matrix-multiply units, but only a small tile of the working set fits at one time.
This asymmetry makes data movement a first-class algorithmic cost. An implementation with the minimum number of floating-point operations can still underperform if it repeatedly materializes large intermediates in HBM.
2.2 Conventional attention
A conventional implementation launches separate kernels for three stages:
The inputs and output require storage, whereas each intermediate requires storage. For long sequences, the quadratic traffic dominates.
For example, with and , each of , , and contains approximately 2.1 million elements. The score matrix contains approximately 268 million elements. The principal problem is not that these scores must be computed; it is that a conventional schedule writes and rereads them.
The consequence is visible without any timing at all. Running the same attention over the same tensors, once with the two intermediates materialized and once through a fused kernel, and recording peak allocated memory:
| Sequence length | Conventional: eager (reference) | Fused: SDPA cuDNN (SM100) | Ratio (eager / cuDNN) |
|---|---|---|---|
| 512 | 56 MB | 46 MB | 1.2× |
| 2,048 | 336 MB | 88 MB | 3.8× |
| 8,192 | 4,368 MB | 256 MB | 17.1× |

This is a memory comparison, not a speed measurement. The widening gap matters more than any one ratio: the conventional path stores two intermediates, while the fused path does not.
2.3 Numerically stable softmax
For one score row , stable softmax is
Subtracting the row maximum prevents exponential overflow. The apparent obstacle to tiling is that both the maximum and normalization sum depend on the complete row.
3. The fused alternative
Tiling either matrix multiplication alone does not solve the problem. A tuned GEMM can efficiently produce , but if is subsequently written to HBM, consumed by a softmax kernel, and written again as , the implementation retains the dominant traffic.
The optimization boundary must therefore include the full operator:
This produces two requirements:
- The score and probability tiles must remain on chip and never become global intermediates.
- Softmax statistics computed from separate key blocks must be merged exactly.
The second requirement is the key algorithmic problem solved by FlashAttention.
4. Design
4.1 Tiled execution
Partition the query matrix into row blocks and the key and value matrices into blocks and . For each query block, the kernel scans all key/value blocks:
- Load , , and into on-chip storage.
- Compute the score tile .
- Merge into the running softmax statistics for .
- Accumulate the weighted values into the running output .
- Discard and proceed to the next key/value block.
Only the final is written to HBM.
4.2 Online softmax
For one query row, maintain three quantities after processing some key blocks:
- : the largest score observed so far;
- : the sum of exponentials relative to ;
- : the unnormalized weighted-value sum relative to .
For a new score block with corresponding value block , compute
The previous state was expressed relative to , whereas the merged state must be expressed relative to . Rescale the old contribution and add the new block:
After all blocks have been processed,
This recurrence is exact in real arithmetic. Every old exponential is converted from the old reference maximum to the new one because
4.3 Worked example
4.4 IO complexity
FlashAttention chooses tile sizes so the active Q, K, V, score, and output tiles fit on chip. The score and probability matrices are never written out in full. The original paper analyzes how much HBM traffic this saves as the on-chip tile capacity grows [1]; the practical point is that the kernel trades reuse of small tiles for the cost of storing two full intermediates.
4.5 Backward pass
For training, saving the full probability matrix for the backward pass would undo that memory saving. FlashAttention instead keeps the output and a normalization statistic per row, then recomputes score and probability tiles as needed during backward. It spends some extra arithmetic to avoid storing quadratic intermediate state [1].
5. Concrete GPU Implementation
5.1 Kernel structure
A simplified forward kernel has the following structure. One thread block owns a query tile, and the online-softmax state is scalar per row. Later versions change the scheduling around this loop, but not the loop itself.
parallel for each query tile Q_i:
load Q_i
m = -infinity
l = 0
z = 0
for each key/value tile (K_j, V_j):
load K_j and V_j into shared memory
S_ij = (Q_i @ transpose(K_j)) / sqrt(d)
new_m = max(m, rowmax(S_ij))
old_scale = exp(m - new_m)
P_ij = exp(S_ij - new_m)
l = old_scale * l + rowsum(P_ij)
z = old_scale * z + P_ij @ V_j
m = new_m
O_i = z / l
store O_i
One thread block typically owns one or more query tiles. Warps cooperate on tensor-core matrix multiplications, row-wise maximum and sum reductions, and movement between HBM, shared memory, and registers. Causal attention skips score tiles that lie entirely above the diagonal and masks only tiles intersecting the diagonal.
5.2 Fusion boundary
The implementation must fuse score computation, masking, softmax, dropout when required, and value accumulation into one kernel. Writing S_ij or P_ij to global memory between these stages defeats the design. The fusion boundary, rather than a novel matrix-multiplication primitive, is what removes the quadratic intermediate traffic.
5.3 Resource tradeoffs
Larger tiles improve data reuse but consume more shared memory and registers, so fewer blocks may run concurrently. Which tile size works best depends on the GPU and the attention shape.
6. What does the performance look like?
The memory plot shows why avoiding intermediates matters; it does not measure runtime. In this separate B300 comparison (, , , BF16, causal), speedup is measured against an eager attention implementation at each sequence length:

The growing speedup is consistent with avoiding quadratic intermediate traffic as sequences get longer. It is not a fixed benefit of “FlashAttention”: short sequences, different head sizes, or a different GPU can change the comparison.
7. What to take away
The two matrix multiplications are not the whole cost of attention. A conventional schedule writes their intermediates to HBM; FlashAttention instead computes and consumes small score tiles on chip. Online softmax makes that possible by carrying a maximum, a normalization sum, and a weighted-value sum from tile to tile.
This changes where data lives, not the dense attention function or its quadratic arithmetic. It also does not promise a fixed speedup: the workload, kernel implementation, and hardware still matter. The next article follows how later GPU implementations schedule this same computation.
References
- Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. NeurIPS 2022.