The KV Cache: Sizing It, and the Attention Variants That Shrink It

Abstract

Autoregressive decoding is cheap only because it does not recompute the past: every generated token reuses the keys and values of every token before it. Those tensors are the KV cache, and its size follows from six numbers with no slack in the derivation. This article works out that formula, shows why the cache — not the weights — sets the batch-size ceiling at long context, and derives the arithmetic intensity that makes decode memory-bound. It then examines the architectural response: multi-query attention, grouped-query attention, and multi-head latent attention are three points on one axis, all of them changing exactly one term in the size formula. It assumes the decoding loop from LLM Inference Introduction.

1. Why the Cache Exists

Decoding generates one token at a time. Producing token $t+1$ requires attention over all previous positions, which needs their keys and values — and those depend only on tokens already fixed, so they never change once computed.

Without a cache, generating $T$ tokens re-runs attention over a growing prefix at every step, doing $O(T^2)$ work in the attention projections alone and re-reading every weight matrix each time. With a cache, each step computes one new key and value, appends them, and attends. The cache turns a quadratic amount of recomputation into a linear amount of storage.

That is the trade the rest of this article is about: the storage is not small.

2. Sizing the Cache

Each layer stores one key vector and one value vector per position per KV head. For a model with $L$ layers, $H_{kv}$ key-value heads, head dimension $d_h$, and $b$ bytes per element, a sequence of $T$ tokens costs

Everything except $T$ is fixed by the architecture, so the useful form is bytes per token:

and total cost is that constant times the number of live tokens, summed over every sequence in the batch.

Worked examples. In BF16 ($b = 2$):

The middle row is the reason grouped-query attention exists. A 70B model in BF16 has 140 GB of weights, which already needs two 80 GB GPUs; with multi-head attention, one 8K-token sequence would add 21.5 GB, and a batch of 32 would need 687 GB of cache — nearly five times the weights. The bottom row is the same model as actually shipped: the cache falls to an eighth, so that batch of 32 needs 86 GB, which one additional GPU covers.

Three properties of the formula are worth stating plainly.

It is linear in context length. Doubling the context doubles the cache. There is no reuse to exploit; every position must be retained.

It is linear in batch size, with no sharing. Two sequences share weights but never share cache — except for a common prefix, which is exactly the case PagedAttention makes exploitable.

It does not depend on the number of query heads. Only $H_{kv}$ appears. That is the loophole the next section exploits.

3. Why Decode Is Memory-Bound

The cache is not just a capacity problem. It is read in its entirety on every decode step, and that read is what sets the speed.

Consider one layer of one decode step. Attention over $T$ cached positions performs $H_q T d_h$ multiply-accumulates for the scores and the same again for the value-weighted sum, so

Their ratio is the arithmetic intensity of attention at decode time:

The context length cancels. What survives is the ratio of query heads to KV heads — the group size — divided by the element width. In BF16 with multi-head attention, where $H_q = H_{kv}$, the intensity is exactly 1 FLOP per byte, against a ridge point of roughly 295 on an H100. Attention at decode time runs at about a three-hundredth of the machine’s arithmetic capability, and no amount of kernel tuning changes that; the operator is reading memory, not computing.

One consequence is easy to get wrong. Increasing the batch size does not help. Batching amortizes weight reads across sequences, which is why it works so well for the feed-forward layers, but every sequence has its own KV cache, so the attention read scales with the batch exactly as the arithmetic does. Attention stays at the same intensity no matter how many requests are in flight.

4. The Variants

Every variant below changes $H_{kv}$, or replaces it with something smaller. Nothing else in the formula is available: $L$ and $d_h$ are the model, $T$ is the request, $b$ is the storage format.

4.1 Multi-head attention (MHA)

The original: $H_{kv} = H_q$, every head with its own projection. Maximum expressiveness, maximum cache. For the 70B model above it is 2.6 MB per token, which is unaffordable at any serious batch size or context length.

4.2 Multi-query attention (MQA)

Proposed in 2019 for exactly this reason: keep all $H_q$ query heads but project a single shared key and value. The cache falls by a factor of $H_q$ — 64× for the model above — and decode intensity rises by the same factor.

The cost is quality. Collapsing to one KV head is a real reduction in what attention can represent, and models trained this way show measurable degradation and, in some reports, training instability. MQA is the right answer when memory is the binding constraint and the quality loss is acceptable.

4.3 Grouped-query attention (GQA)

The interpolation, and now the default. Partition the $H_q$ query heads into $H_{kv}$ groups; every head in a group shares one key-value pair. At $g = 8$, the cache is one eighth of MHA and the intensity is eight times better, with quality close to MHA.

Two practical properties explain its adoption. First, the natural choice is $H_{kv}$ equal to the tensor-parallel degree, so each GPU holds exactly one KV head and no cache is replicated across devices. Second, an existing MHA checkpoint can be converted rather than retrained: mean-pool the key and value projections within each group and continue training for a small fraction of the original budget. Llama 2’s 70B model, Mistral, and most models released since use GQA.

4.4 Multi-head latent attention (MLA)

A different move. Instead of reducing the number of KV heads, compress keys and values jointly into a low-rank latent vector and cache that, reconstructing per-head keys and values on the fly. The cached object is one vector of dimension $d_c$ per token per layer, with $d_c$ far smaller than $H_{kv} d_h$, plus a small decoupled component carrying rotary position information that cannot be compressed away.

DeepSeek-V2 reports a cache about 93% smaller than its multi-head predecessor at comparable quality. The trade is arithmetic: the reconstruction is extra work on every step, and the kernel is more complex than either MHA or GQA.

5. The Other Levers

Changing the architecture is not the only way to shrink the cache, and the alternatives compose with it.

Quantize it. The cache is a tensor like any other, and quantizing it to INT8 or INT4 halves or quarters it directly. It has its own asymmetry: keys carry channel-wise outliers and want per-channel scales, while values are well behaved under per-token scaling. At long context this is often the single largest win available, because it is orthogonal to everything else in this article.

Bound what is kept. Sliding-window attention caps the cache at the window size instead of the context length, making it constant in $T$ rather than linear. Attention-sink methods keep the first few positions permanently alongside a recent window, recovering most of the quality that a naive window loses. Eviction policies go further and drop positions judged unimportant.

Share across layers. Adjacent layers can be made to share one set of keys and values, dividing the cache by the sharing factor along $L$ — the same idea as GQA applied to the layer axis rather than the head axis.

Stop wasting what you allocate. PagedAttention does not make the cache smaller; it removes the fragmentation and over-reservation that made the allocated cache several times the used cache, and lets sequences with a common prefix share physical blocks.

6. What This Means for a Serving System

Three consequences follow from the formula and the intensity result.

The cache sets the batch-size ceiling at long context. Weights are a fixed cost paid once; cache is a per-sequence cost that grows with the context. Past a few thousand tokens the cache is the term that decides how many requests fit, which is why throughput at 32K context looks nothing like throughput at 512.

Decode speed tracks cache bytes, not FLOPs. Since attention at decode is memory-bound with intensity $2g/b$, anything that reduces bytes read per token — a larger group size, a compressed latent, a quantized cache, a shorter window — translates almost directly into tokens per second. Anything that only reduces arithmetic does not.

Prefill and decode want different things. Prefill processes many tokens at once and is compute-bound; decode processes one and is bandwidth-bound. The KV cache is written in the first phase and read in the second, which is why serving systems increasingly schedule the two separately, and sometimes on different hardware.

7. Summary

• KV cache size is $2 L H_{kv} d_h T b$ bytes, linear in context and batch, and independent of the number of query heads.
• Decode attention has arithmetic intensity $2g/b$ where $g = H_q/H_{kv}$ — one FLOP per byte for multi-head attention in BF16, roughly 300× below the hardware’s ridge point. Batching does not improve it.
• MQA, GQA, and MLA are the same move at different strengths: reduce what has to be stored per token. GQA at 8 groups is the current default because it recovers most of the memory at a small and measurable quality cost, and converts cleanly from an existing MHA checkpoint.
• Quantization, windowing, cross-layer sharing, and paged allocation are orthogonal levers that compose with the architectural choice.

References

1. Noam Shazeer. Fast Transformer Decoding: One Write-Head is All You Need. 2019.
2. Joshua Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023.
3. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024.
4. Reiner Pope et al. Efficiently Scaling Transformer Inference. MLSys 2023.
5. Woosuk Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
6. Guangxuan Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR 2024.
7. Zirui Liu et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML 2024.