Introduction

Autoregressive decoding avoids recomputing the past by keeping earlier tokens’ keys and values in a KV cache. The optimizations in this article can be organized around one storage formula:

KV-cache payload bytes≈U×Lcache×R×b \boxed{\text{KV-cache payload bytes} \approx U \times L_{\text{cache}} \times R \times b}
  • UU: how many token positions are stored. Count positions across all requests, counting a physically shared prefix only once.
  • LcacheL_{\text{cache}}: how many distinct layer states are stored per position. Normally this is the number of layers.
  • RR: how many elements represent each position in each cached layer state. This is where head sharing and latent compression act.
  • bb: how many bytes store each element. This is where KV quantization acts.

This is a payload model for uniform cached records, not a count of padding, allocator waste, quantization metadata, or workspace. With unequal layer formats or retention policies, sum the corresponding payloads instead of using one product.

We will map each optimization to a factor, then separate storage savings from speed: the measured head-sharing sweep reduces bytes eightfold but latency by only about 2.4×. The detailed sections explain the mechanisms behind that difference. This article assumes the decoding loop from LLM Inference Introduction.

1. Why the Cache Exists

Decoding generates one token at a time. Producing token t+1t+1 requires attention over all previous positions, which needs their keys and values, and those depend only on tokens already fixed, so they never change once computed.

The cache therefore avoids recomputing past attention, trading a linear amount of storage for quadratic rather than cubic total attention work.

That is the trade the rest of this article is about: the storage is not small.

2. Sizing the Cache

First recover the familiar formula from the four factors. For BB independent sequences of length TT, U=BTU = BT. Without cross-layer sharing, Lcache=LL_{\text{cache}} = L. Ordinary attention stores a key and a value for every KV head, so R=2HkvdhR = 2 H_{kv} d_h, where HkvH_{kv} is the KV-head count and dhd_h the head dimension. Thus one sequence costs

KV bytes  =  2⏟K and V⋅L⋅Hkv⋅dh⋅T⋅b. \text{KV bytes} \;=\; \underbrace{2}_{K \text{ and } V} \cdot L \cdot H_{kv} \cdot d_h \cdot T \cdot b .

For a fixed architecture and storage precision, the useful form is bytes per token:

bytes/token  =  2⋅L⋅Hkv⋅dh⋅b, \text{bytes/token} \;=\; 2 \cdot L \cdot H_{kv} \cdot d_h \cdot b ,

and total cost is that constant times the number of live tokens, summed over every sequence in the batch.

The optimization map. Each row changes part of the same product:

OptimizationFactorWhat changes
Multi-head attention (MHA)RRBaseline: R=2HqdhR = 2 H_q d_h, with one KV head per query head.
Grouped-query attention (GQA)RRShare KV heads: R=2(Hq/g)dhR = 2(H_q/g)d_h.
Multi-query attention (MQA)RRShare one KV head across all queries: R=2dhR = 2d_h.
Multi-head latent attention (MLA)RRChange the cached representation; the DeepSeek-V2 format below uses R=dc+drR = d_c + d_r.
KV quantizationbbUse fewer bits per element, allowing for scales and other metadata.
Sliding windows or evictionUURetain fewer token positions, with model and quality constraints.
Cross-layer KV sharingLcacheL_{\text{cache}}Store fewer distinct layer states where the architecture supports sharing.
Prefix sharingUUStore an identical prefix once for multiple requests.

Paging acts around the product: it reduces allocation waste rather than shrinking each token’s record. Prefix sharing can additionally reduce the stored payload. These are different benefits.

Worked examples. In BF16 (b=2b = 2):

ModelLLHkv×dhH_{kv} \times d_hBytes/token8K context×32 sequences
7B, multi-head3232×12832 \times 128512 KB4.3 GB137 GB
70B, multi-head (hypothetical)8064×12864 \times 1282.5 MB21.5 GB687 GB
70B, grouped-query (8 groups)808×1288 \times 128320 KB2.7 GB86 GB

The middle row is the reason grouped-query attention exists. A 70B model in BF16 has 140 GB of weights, which already needs two 80 GB GPUs; with multi-head attention, one 8K-token sequence would add 21.5 GB, and a batch of 32 would need 687 GB of cache, nearly five times the weights. The bottom row is the same model as actually shipped: the cache falls to an eighth, so that batch of 32 needs 86 GB, which one additional GPU covers.

Three properties of the formula are worth stating plainly.

It is linear in context length. Doubling the context doubles the cache. There is no reuse to exploit; every position must be retained.

It is linear in batch size, with no sharing. Two sequences share weights but never share cache, except for a common prefix, which is exactly the case PagedAttention makes exploitable.

It does not depend on the number of query heads. Only HkvH_{kv} appears. That is the loophole the next section exploits.

The 512 KB figure in the first row is not just arithmetic. Allocating one layer’s cache for one token as a (2, 1, 32, 128) BF16 tensor gives 16,384 bytes, and multiplying by 32 layers gives 524,288 bytes, exactly 512 KB. Every capacity number later in this series is built from that constant, so it is worth confirming once that the formula and the tensor layout agree.

3. Why Decode Is Memory-Bound

The product ULcacheRbU L_{\text{cache}} R b explains capacity, not latency. For dense decode, reading the retained history often dominates attention time. The memory-transfer component is approximately

tKV read≈bytes actually fetchedachieved bandwidth. t_{\text{KV read}} \approx \frac{\text{bytes actually fetched}}{\text{achieved bandwidth}} .

A smaller stored payload need not reduce actual traffic or execution time by the same factor. Reuse, tile reloads, dequantization, latent-attention computation, and kernel scheduling all matter. A shared prefix is stored once but can still be read for multiple requests; batching can improve achieved bandwidth without shrinking any record.

Consider one layer of one decode step. Attention over TT cached positions performs HqTdhH_q T d_h multiply-accumulates for the scores and the same again for the value-weighted sum, so

FLOPs=4 Hq T dh,bytes read=2 Hkv T dh b. \text{FLOPs} = 4\,H_q\,T\,d_h , \qquad \text{bytes read} = 2\,H_{kv}\,T\,d_h\,b .

Their ratio is the arithmetic intensity of attention at decode time:

Idecode=4HqTdh2HkvTdhb=2b⋅HqHkv. I_{\text{decode}} = \frac{4 H_q T d_h}{2 H_{kv} T d_h b} = \frac{2}{b}\cdot\frac{H_q}{H_{kv}} .

The context length cancels. What survives is the ratio of query heads to KV heads, the group size, divided by the element width. In BF16 with multi-head attention, where Hq=HkvH_q = H_{kv}, the intensity is exactly 1 FLOP per byte, against a ridge point of roughly 295 on an H100 (2,000 TFLOP/s BF16 over 3.35 TB/s HBM). Attention at decode time runs at about a three-hundredth of the machine’s arithmetic capability, and no amount of kernel tuning changes that; the operator is reading memory, not computing. The ratio 2g/b2g/b is hardware-independent; only the ridge point it is compared against changes from one GPU to the next.

3.1 Why prefill does more work per byte

The attention math is the same: compute QK⊤QK^\top, apply softmax, and multiply by VV. What changes is the number of query positions processed together within one sequence, not simply the number of requests in a batch. Decode processes one new query; prefill processes many.

For a four-token prompt, causal prefill computes 1+2+3+4=101 + 2 + 3 + 4 = 10 query-key pairs. A decode step processing only the fourth position computes four. Both use the same four positions’ keys and values, but prefill uses the earlier positions repeatedly for different queries. A tiled kernel can load a KV tile into on-chip memory and reuse it across several query rows before discarding it, doing more arithmetic per byte fetched from HBM.

Batching independent decode requests does not provide the same KV reuse, because each request has its own cache; short prefill workloads may also remain limited by launch overhead or insufficient parallelism.

3.2 Measured intensity and the effect of batching

The decode ratio is measurable rather than merely derivable. On a B300 (2,250 TFLOP/s BF16, 8 TB/s HBM, so a ridge point of 281 FLOP/byte), a paged decode kernel with Hq=Hkv=8H_q = H_{kv} = 8 and dh=128d_h = 128 computes at an arithmetic intensity of 1.0 FLOP/byte at every batch size and context length tested, and reaches at most 75% of HBM peak, while prefill at the same head dimension reaches 78% of compute peak.

OperationShapeTFLOP/sGB/sFLOP/byte% of peak
prefillL=16,3841,7534284,09677.9% (compute)
decodeB=1, kv=4,0960.54641.05.8%
decodeB=32, kv=4,0965.45,4001.067.5%
decodeB=32, kv=16,3846.06,0131.075.2%
B300 roofline at 16,384 tokens: decode batches 1, 8, and 32 remain at 1 FLOP per byte while achieving 1.838, 5.621, and 6.013 TFLOP/s. The reported prefill point reaches 1,753 TFLOP/s at 4,096 FLOP per byte; its batch size is unspecified.
Figure 1. Batch size changes how closely decode approaches the memory roof, not its arithmetic intensity. All three decode points use a 16,384-token context; their throughput is derived from the documented bandwidth below multiplied by 1 FLOP/byte. The prefill point uses the reported values above. This replot uses rounded values transcribed from the post, not recovered original raw measurements.

The figure data and plotting script are stored beside the image. From the repository root, run python content/posts/llm-inference-intro/figs/kv-cache-and-attention-variants/plot_roofline.py to regenerate it using the shared acm_style.py helpers.

What batching does change is whether the memory system is actually saturated. Those are different statements, and the distinction matters. Measured on the same kernel, at a context length of 16,384:

BatchArithmetic intensity (FLOP/byte)Achieved read bandwidth% of HBM peakPer-request cost
11.01,838 GB/s23%583 µs
81.05,621 GB/s70%77 µs
321.06,013 GB/s75%11 µs

Intensity never leaves 1.0 FLOP/byte, yet throughput improves 51× per request. The intensity fixes the ceiling, you can never escape the memory roof, while the batch size decides how close to that ceiling you get. A single sequence leaves the HBM channels mostly idle; thirty-two concurrent sequences keep them busy. This is why serving throughput is a scheduling problem, not just a kernel problem.

Log-log plot of achieved read bandwidth against batch size for two context lengths, rising from about a quarter of peak at batch one towards three quarters at batch thirty-two, with the HBM peak drawn as a horizontal reference line.
Figure 2. Achieved read bandwidth against batch size, with the HBM peak drawn for reference. The same kernel, at the same arithmetic intensity, moves three times more of the machine's bandwidth at batch 32 than at batch 1. Intensity sets the roof; occupancy of the memory system decides how close you run to it.

4. The HkvH_{kv} axis: MHA, MQA, GQA

Which factor changes? RR, the elements stored per token per layer. Hold token count, layer count, head dimension, and precision fixed. MHA, GQA, and MQA retain explicit keys and values but change how many KV heads share the query workload:

RMHA=2Hqdh,RGQA=2Hqgdh,RMQA=2dh. R_{\text{MHA}} = 2 H_q d_h,\qquad R_{\text{GQA}} = 2 \frac{H_q}{g} d_h,\qquad R_{\text{MQA}} = 2 d_h .

All three use the same storage product. MLA also changes RR, but replaces the explicit KV-head representation rather than choosing another point on this head-sharing axis.

Eight query heads mapped to eight key-value heads under multi-head attention, to two under grouped-query attention, and to one under multi-query attention.
Figure 3. The same eight query heads under three schemes. Multi-head attention gives each query head private keys and values; multi-query attention gives all of them one shared pair; grouped-query attention partitions them into groups that share within the group. Only the boxes in the lower row are stored per token, so the cache shrinks in proportion to how many there are.

4.1 Multi-head attention (MHA)

Every query head gets its own key and value. Eight query heads, eight KV heads, eight caches per token:

  q0   q1   q2   q3   q4   q5   q6   q7      query heads (H_q = 8)
   |    |    |    |    |    |    |    |
  k0   k1   k2   k3   k4   k5   k6   k7      KV heads  (H_kv = 8)
   |    |    |    |    |    |    |    |
  c0   c1   c2   c3   c4   c5   c6   c7      stored per token

  group size g = 1        KV heads stored = 8        intensity = 2/b

The original: Hkv=HqH_{kv} = H_q, every head with its own projection. Maximum expressiveness, maximum cache. For the 70B model above it is 2.5 MB per token, which is unaffordable at any serious batch size or context length.

4.2 Multi-query attention (MQA)

Keep all eight query heads, share a single key and value among them:

  q0   q1   q2   q3   q4   q5   q6   q7      query heads (H_q = 8)
   |    |    |    |    |    |    |    |
   +----+----+----+----+----+----+----+---->
                        |
                       k0                    KV heads  (H_kv = 1)
                        |
                       c0                    stored per token

  group size g = 8        KV heads stored = 1        intensity = 16/b

Proposed in 2019 for exactly this reason: keep all HqH_q query heads but project a single shared key and value. The cache falls by a factor of HqH_q, 64× for the model above, and decode intensity rises by the same factor.

The cost is quality. Collapsing to one KV head is a real reduction in what attention can represent, and models trained this way show measurable degradation and, in some reports, training instability. MQA is the right answer when memory is the binding constraint and the quality loss is acceptable.

4.3 Grouped-query attention (GQA)

The interpolation, and now the default. Partition the query heads into groups; every head within a group shares one key-value pair. With two groups:

  q0   q1   q2   q3   q4   q5   q6   q7      query heads (H_q = 8)
   |    |    |    |    |    |    |    |
   +----+----+    |    |    +----+----+
        |         |    |         |
       k0         |    |        k1           KV heads  (H_kv = 2)
        |         |    |         |
       c0                     c1             stored per token

  group size g = 4        KV heads stored = 2        intensity = 8/b

  g = 1  <-----------------+-----------------> g = H_q
     MHA                  GQA                   MQA
                        (g = 8 typical)

At g=8g = 8, the cache is one eighth of MHA and the intensity is eight times better, with quality close to MHA.

Two practical properties explain its adoption. First, the natural choice is HkvH_{kv} equal to the tensor-parallel degree, so each GPU holds exactly one KV head and no cache is replicated across devices. Second, an existing MHA checkpoint can be converted rather than retrained: mean-pool the key and value projections within each group and continue training for a small fraction of the original budget. Llama 2’s 70B model, Mistral, and most models released since use GQA.

4.4 The variants side by side

VariantCached per token per layerRelative cacheDecode intensityChoose when
MHA2Hqdhb2 H_q d_h b112/b2/bcontexts are short and quality dominates
GQA, group size gg2(Hq/g)dhb2 (H_q/g) d_h b1/g1/g2g/b2g/bbalance quality, cache size, and tensor-parallel layout
MQA2dhb2 d_h b1/Hq1/H_q2Hq/b2H_q/bmemory is the binding constraint and you can retrain

MLA is deliberately absent from that table; it is not on this axis and gets its own section next.

The formulas are worth applying to one concrete model rather than three separate ones. The 70B shape from §2 (L=80L = 80, Hq=64H_q = 64, dh=128d_h = 128, BF16) at 8K context:

VariantHkvH_{kv}Cache/tokenOne 8K sequence×32 sequencesIntensity
MHA642.5 MB21.5 GB687 GB1.0
GQA, g=8g{=}88320 KB2.7 GB86 GB8.0
MQA140 KB335 MB10.7 GB64.0

Going from MHA to GQA divides RR by eight and makes the payload eight times smaller. The last column describes a different quantity: arithmetic intensity rises eightfold because the query workload remains while fewer KV bytes are read. This creates an opportunity for lower latency, not a guarantee of proportional speedup.

But only up to a point, and the limit is worth stating here rather than leaving to the measurement section: an eight-times reduction in bytes does not buy an eight-times reduction in latency. Measured, it buys about 2.4×.

Bars comparing kilobytes of KV cache per token for multi-head, eight-group grouped-query, multi-query, and latent attention on an eighty-layer model.
Figure 4. Cache per token for an 80-layer, 64-head model in BF16, at $d_h = 128$. The vertical axis of the problem is capacity and bandwidth at once: each bar is simultaneously what a token costs to store and what it costs to re-read on every subsequent decode step.

4.5 What the variants do to the weights

Everything so far has been about the cache. The variants also change the model’s parameter shapes, and they do so asymmetrically, which is the part that is easy to get wrong. Only the key and value projections shrink. The query and output projections keep their full width in every variant.

With D=HqdhD = H_q d_h the model width:

ProjectionMHAGQA, group size ggMQA
WQW_QD×DD \times DD×DD \times DD×DD \times D
WKW_KD×DD \times DD×DgD \times \frac{D}{g}D×dhD \times d_h
WVW_VD×DD \times DD×DgD \times \frac{D}{g}D×dhD \times d_h
WOW_OD×DD \times DD×DD \times DD×DD \times D

The reason is structural rather than incidental. There are always HqH_q query heads, so WQW_Q must emit Hqdh=DH_q d_h = D columns, and WOW_O must consume DD and return DD. Only the K and V projections emit one column block per KV head, and there are fewer of those.

It is worth being precise here because the phrasing “each head has its own WQW_Q, WKW_K, WVW_V” invites a wrong mental picture. In code there is one matrix, and the heads are slices of its output:

# MHA: Q, K, V all project to D columns
self.W_Q = nn.Linear(D, D)
self.W_K = nn.Linear(D, D)
Q = self.W_Q(X).view(B, L, H_q, d_h).transpose(1, 2)   # heads = column blocks

# GQA: only K and V get narrower
self.W_K = nn.Linear(D, H_kv * d_h)          # D/g columns worth
K = self.W_K(X).view(B, L, H_kv, d_h).transpose(1, 2)

Splitting one (D,D)(D, D) weight into HqH_q column blocks is exactly equivalent to having HqH_q separate (D,dh)(D, d_h) matrices, because a linear layer’s output columns are independent. It is also why production stacks fuse the three projections into one qkv_proj: one GEMM beats three.

Applied to the 70B shape, the asymmetry shows up as a real parameter difference:

VariantWKW_K shapeAttention params/layer× 80 layers
MHA (hypothetical)8192×81928192 \times 8192268 M21.5 B
GQA, g=8g{=}8 (as shipped)8192×10248192 \times 1024151 M12.1 B
MQA8192×1288192 \times 128136 M10.9 B

GQA saves 9.4 billion parameters against multi-head, roughly 13% of a 70B model. That is a larger effect than it first appears, because attention is being compared against itself; the FFN in the same layer is 704 M parameters, so GQA attention (151 M) is about a fifth of the block.

One last consequence is practical, and it explains why the checkpoint-conversion trick in §4.3 works at all. Moving down this table is a reshape: going from MHA to GQA discards columns within each group and averages what remains, which is exactly the mean-pooling operation used to initialise from an existing checkpoint. Moving up, or going to MQA, is not a reshape (87% of WKW_K’s columns have no counterpart), so those models are trained from scratch. Shape compatibility runs one way only.

5. Compressing instead of dropping heads: MLA

Which factor changes? Again RR, but by changing the record rather than sharing explicit KV heads. GQA stores fewer sets of per-head keys and values. MLA instead caches a compact latent representation from which the attention computation obtains the information it needs. It should not be modeled as keeping HkvH_{kv} fixed and compressing each cached head independently.

In the DeepSeek-V2 format, one token’s cached record contains a joint KV latent of dimension dcd_c plus a decoupled positional component of dimension drd_r [3]. In the shared formula, replace the ordinary record size with

RMLA=dc+dr,KV-cache payload bytes≈ULcache(dc+dr)b. R_{\text{MLA}} = d_c + d_r,\qquad \text{KV-cache payload bytes} \approx U L_{\text{cache}} (d_c + d_r) b .

DeepSeek-V2 makes those dimensions concrete: dc=512d_c = 512 compressed plus dr=64d_r = 64 decoupled RoPE. That is 576 elements per token per layer, against 2Hkvdh=2⋅128⋅128=32,7682 H_{kv} d_h = 2 \cdot 128 \cdot 128 = 32{,}768 for an equivalent multi-head layer, about 1.8% of the naive per-layer cache. The paper reports roughly a 93% end-to-end reduction; the two figures measure different baselines, since the end-to-end number compares against a sibling model with its own layer count and head configuration rather than against a like-for-like MHA layer. This calculation establishes storage savings, not a latency ratio.

  stored per token, per layer:

     MHA                     MLA
  +---------+            +---------+
  |   k0    |            |         |
  |   k1    |            |  latent |  d_c = 512   compressed K and V
  |   ...   |   ---->    |         |
  | k_{127} |            +---------+
  |   v0    |            |  rope   |  d_r = 64    cannot be compressed
  |   ...   |            +---------+
  | v_{127} |
  +---------+            576 elements
  32,768 elements

  Explicit per-head K and V are replaced by one compact record.

The query heads still exist, but the cached object is no longer a list of independent KV-head vectors. That is why the group-size formula 2g/b2g/b cannot simply be applied to MLA.

The computation changes too. MLA uses down- and up-projections rather than only narrowing WKW_K and WVW_V. DeepSeek-V2 describes absorbing the KV up-projections into the query and output projections for inference, avoiding explicit reconstruction of the full cached K and V [3]. Actual speed depends on that execution path and its kernels. A smaller RR does not by itself establish either fewer total model parameters or faster decoding.

6. Measuring the Variants

The storage formula predicts exactly how head sharing changes RR. A proportional latency reduction would additionally require unchanged achieved bandwidth and no other dominant cost. The following sweep tests that extra assumption.

Sweep Hkv∈{8,4,2,1}H_{kv} \in \{8, 4, 2, 1\} with Hq=8H_q = 8 fixed, so the group size takes the values a real model would choose: g=1g = 1 is MHA, g=8g = 8 is MQA. Same kernel, same page size, same dtype; only the head axis of the cache changes. At a 16K context and batch 32:

HkvH_{kv}ggbytes read per steplatencyspeedupachieved GB/s% of HBM peak
8 (MHA)12048 MB0.360 ms1.00×5,96674.6%
421024 MB0.214 ms1.68×5,00862.6%
24512 MB0.177 ms2.04×3,03738.0%
1 (MQA)8256 MB0.152 ms2.38×1,77222.1%

Bytes fall by exactly the factor the formula predicts: eight. Latency falls by 2.38×. The prediction is wrong by a factor of 3.4, and it is not wrong because the machine was idle. Repeating the sweep at batch 64 and 64K context, where the baseline already sustains 88.9% of peak, still yields only 2.12×.

Log-log plot: the dashed prediction line rises linearly with group size to eight times, while the measured curves flatten at roughly two to two and a half times for every context length and batch size.
Figure 5. Measured speedup against group size, with the prediction from the byte formula drawn as a straight line. The two agree at $g=1$ by construction and diverge immediately: the measured curve flattens near 2.4× where the prediction reaches 8×. Every batch size and context length tested produces the same flattening.
Plot of achieved bandwidth against group size, falling steeply from about three quarters of peak at group size one to under a quarter at group size eight.
Figure 6. The mechanism. As the group size grows, the bytes a decode step must read fall eightfold, but the fraction of HBM peak the kernel can sustain falls with them, from 75% to 22%. The cache gets smaller faster than the memory system can exploit the reduction.

The obvious explanation, that narrower reads coalesce worse, turns out to be wrong, and it is worth showing the falsification rather than asserting the answer. In the standard NHD layout the contiguous run for one token is HkvdhbH_{kv} d_h b bytes, so the two candidates are access granularity and parallelism. Both can be varied independently.

Hold the head count at one and widen the contiguous run by raising dhd_h, and bandwidth does not move at all:

dhd_hcontiguous runachieved GB/s% of peak
64128 B1,64720.6%
128256 B1,77922.2%
256512 B1,74721.8%
5121024 B1,73921.7%

An eight-times wider slice changed bandwidth by one percentage point, while latency tracked bytes exactly. Access granularity is not the cause. Nor is parallelism: at fixed Hkv=1H_{kv}=1, going from 16 to 512 concurrent requests, 32× more independent streams, moved bandwidth only from 19.7% to 24.0%.

Log-log plot of latency against bytes read per decode step, showing bytes falling eightfold while latency falls only about twofold.
Figure 7. Bytes against latency, both on logarithmic axes. If the byte formula predicted latency the points would lie on a line of slope one; the measured slope is visibly shallower. The horizontal distance between the first and last point is the eight-fold byte saving, and the vertical distance is the 2.4× latency saving actually obtained.

Return to the product: the sweep reduces RR, but keeps the query-head count and context fixed. The leading score and value arithmetic therefore stays roughly constant while bytes fall; work per byte rises with gg. Non-memory costs can consequently become more visible. Identifying the specific bottleneck would require instruction-level profiling, not just effective-bandwidth measurements.

Three consequences follow, and they are more useful than the prediction was.

A smaller RR has diminishing latency returns in this sweep. The last doubling of gg from 4 to 8 bought 17% of latency while costing more than half the bandwidth efficiency. That is a workload-specific trade-off, not a universal optimum at g=8g = 8.

MLA needs its own performance measurement. Its smaller latent record reduces RR, but also changes the computation. The MHA/GQA sweep cannot predict MLA’s work per byte or guarantee that it avoids the same latency plateau.

Different factors have different runtime consequences. Quantization reduces bb but adds conversion work; a window reduces UU and the number of attended positions; sharing cached state saves storage without necessarily removing each consumer’s reads. The product tells us what to count, while measurements establish what gets faster.

7. The Other Levers

Return to ULcacheRbU L_{\text{cache}} R b. Head sharing and MLA change RR; the remaining levers change how many records are kept, how often they are stored, or how their elements are encoded. They can compose where the model and serving engine support them.

Quantize it: reduce bb. The cache is a tensor like any other. Quantizing it from BF16 to an 8-bit format halves the element payload; a packed 4-bit format quarters it. The 512 KB/token example becomes 256 KB/token or 128 KB/token before scales, zero points, padding, and any higher-precision residuals. The representation size RR need not change.

This also illustrates composition with MLA: its 576-element record occupies 576×2=1152576 \times 2 = 1152 payload bytes in BF16, or 576 bytes in an 8-bit encoding, if supported. These are storage calculations, not measured speedups or quality guarantees. Quantization adds conversion work and can affect quality. KIVI finds that keys benefit from per-channel quantization while values benefit from per-token quantization [7], so metadata and scaling choices matter.

For ordinary MHA, the idealized intensity 2/b2/b rises from 1 to 2 FLOP/byte when element width halves. That is more work per byte, not constant work per byte. Whether latency improves proportionally depends on conversion overhead and achieved bandwidth.

Bound what is kept: reduce UU. For BB independent requests with a supported window of WW tokens, U=Bmin⁡(T,W)U = B\min(T,W) rather than BTBT. Attention-sink methods retain a small initial prefix alongside recent tokens [6]; eviction policies discard selected positions. These change which history remains available and require model compatibility or quality evaluation, unlike merely storing the same elements at lower precision.

Share across layers: reduce LcacheL_{\text{cache}}. If an architecture lets each set of ss layers share one cached state, Lcache=L/sL_{\text{cache}} = L/s for equal-size groups. This reduces the number of stored records, not the number of Transformer layers or necessarily the number of reads. It is an architectural feature, not an arbitrary inference-time substitution.

Share prefixes: reduce physical UU. If BB requests share an identical reusable prefix of PP tokens and each has a private continuation of CC tokens, separate storage needs U=B(P+C)U = B(P+C) positions; sharing needs only U=P+BCU = P+BC. Each request still performs its own attention, so storage sharing alone does not guarantee shared reads.

Stop wasting what you allocate: reduce overhead around the product. PagedAttention allocates cache blocks on demand rather than reserving a request’s maximum future length [5]. It reduces over-reservation and fragmentation, though a partly filled final block can still waste space. Prefix sharing is an additional way to reduce the payload itself. Neither mechanism changes RR or bb.

8. What This Means for a Serving System

Use the formula as a checklist: which factor changes, what does the engine actually allocate and read, and which runtime cost remains?

The cache sets the batch-size ceiling at long context. Weights are a fixed cost paid once; cache is a per-sequence cost that grows with the context. Past a few thousand tokens the cache is the term that decides how many requests fit, which is why throughput at 32K context looks nothing like throughput at 512.

Payload savings and speedups are separate results. Use ULcacheRbU L_{\text{cache}} R b to estimate stored bytes, then evaluate actual traffic, achieved bandwidth, computation, and conversion overhead. In §6, reducing RR eightfold gives about 2.4× lower latency. That result does not predict the speedup from quantizing bb, changing MLA’s record format, or sharing physical prefixes.

Prefill and decode want different things. Prefill processes many query positions together and can become compute-bound; decode processes one new query per request and often depends on KV-read bandwidth. Batching improves device occupancy and bandwidth use without shrinking each record. It optimizes execution around the storage formula rather than changing the record size.

9. Summary

  • Start with ULcacheRbU L_{\text{cache}} R b: stored token positions, distinct cached layer states, elements per record, and bytes per element.
  • MHA, GQA, and MQA change RR through KV-head sharing. MLA changes RR by storing a latent record instead of explicit per-head K and V.
  • Quantization reduces bb; windows and prefix sharing reduce UU; cross-layer sharing reduces LcacheL_{\text{cache}}. Paging reduces allocation overhead around the payload.
  • Storage savings are not latency guarantees. In the measured head-sharing sweep, bytes fall eightfold but latency improves about 2.4× as achieved bandwidth drops. MLA and quantization require their own measurements.
  • Batching optimizes execution. It can improve bandwidth use at unchanged decode intensity without shrinking the persistent KV record.

References

  1. Noam Shazeer. Fast Transformer Decoding: One Write-Head is All You Need. 2019.
  2. Joshua Ainslie et al. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints. EMNLP 2023.
  3. DeepSeek-AI. DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. 2024.
  4. Reiner Pope et al. Efficiently Scaling Transformer Inference. MLSys 2023.
  5. Woosuk Kwon et al. Efficient Memory Management for Large Language Model Serving with PagedAttention. SOSP 2023.
  6. Guangxuan Xiao et al. Efficient Streaming Language Models with Attention Sinks. ICLR 2024.
  7. Zirui Liu et al. KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache. ICML 2024.