The KV cache

01 · Concept

Concept

Suppose you serve Qwen3.8-27B and a user opens a conversation that grows toward the model’s native 262,144-token context. The weights are a fixed cost. What is not fixed is the memory that conversation holds hostage while it is active — and to size it, you need to know exactly what the KV cache is and, for this model, which layers actually keep one.

A decoder-only model generates one token, appends it, and repeats. A naive implementation would rerun the entire network over the growing sequence at every step, redoing identical work on the prefix. The KV cache stores the attention states future tokens need, turning repeated computation into reusable memory. In one attention layer, every token produces query, key, and value projections. During causal decode only the newest position needs a new output: its query attends to cached keys and mixes cached values, while past queries are never needed again. So the cache retains past keys and values, and the new token computes only its own $q_t$, $k_t$, and $v_t$. For one head,

where the cached matrices grow as $k_t$ and $v_t$ are appended. This is a consequence of exact causal attention, not an approximation to it. Ignoring allocator overhead, the memory model is

with the factor two for keys plus values, $L$ cached layers, $T$ positions, $H_{kv}$ KV heads, $D$ head dimension, and $B$ bytes per element.

Now plug in Qwen3.8-27B — carefully, because there is a classic wrong turn here. The config says 64 layers, 4 KV heads, head dimension 256, bf16. Per token per layer: $4 \times 256 \times 2 \times 2\ \text{B} = 4096\ \text{B} = 4\ \text{KiB}$. Multiply by 64 layers and you get 256 KiB per token, and at full context $256\ \text{KiB} \times 262{,}144 = 64\ \text{GiB}$ for a single sequence — which, next to 54 GB of bf16 weights, would fill a 192 GB accelerator to the brim at two users and overflow it at three. That estimate is wrong, and wrong in an instructive way: it assumes every layer is an attention layer.

Lesson 4.16 showed that only 16 of the 64 layers are full attention; the other 48 are Gated DeltaNet layers, which keep a fixed-size recurrent state and cache nothing per token. The correct derivation:

The wrong assumption quadrupled the answer. Sixteen gibibytes is still enormous — about a fifth of an 80 GB card for one conversation — but it is the real number, and every capacity plan in tracks 8 and 9 builds on it.

The 48 DeltaNet layers are the counterpoint. The reference Transformers implementation keeps the recurrent matrix state in float32: 48 value heads × 128 × 128 × 4 bytes = 3 MiB per layer, or 144 MiB across all 48, excluding a much smaller short-convolution state. That figure is constant in sequence length. Token one and token 262,144 cost the matrix state the same memory. Other runtimes may choose a different state dtype, so inspect the realized allocation; the architectural bargain is the flat shape, not a universal byte count.

Even within the caching layers, the 4 KV heads are themselves a saving: the model asks 24 query heads to share 4 K/V heads (grouped-query attention), shrinking $H_{kv}$ and therefore cache size and decode bandwidth. Lesson 7.6 compares this choice against MQA and MLA numerically. Quantized cache formats shrink $B$; sliding windows shrink effective $T$ at the price of forgetting.

Sizing is only half the engineering. Decode reads a long history to produce little new computation, so moving K/V data can dominate; kernels need friendly layouts while sequences grow unpredictably. Reserving every request’s maximum 16 GiB up front would be ruinous; paged allocation hands out fixed-size blocks on demand instead (lesson 7.8). Prefix caching shares blocks between requests that begin with the exact same token sequence, such as a common system prompt — safe only under the same model, adapter, and positional treatment, and never across authorization boundaries.

One last correction to the intuition: the cache does not make decode constant-time in context. Prefix re-projection disappears, but each new query in the 16 attention layers still reads and attends over everything retained. The durable model is a transcript — computation already performed becomes memory, saving arithmetic at the price of growing bandwidth, capacity, lifecycle, and privacy responsibilities — kept, in this model, by only a quarter of the layers.