The KV cache

1

Concept

A decoder-only language model generates one token, appends it to the context, and repeats. A naive implementation would run the entire Transformer over the growing sequence on every step. Most work on earlier positions would be identical. The KV cache stores the attention states that future tokens need, turning that repeated prefix computation into reusable memory.

In one attention layer, every token produces query, key, and value projections. During causal decode, only the newest position needs a new output. Its query can attend to keys from all allowed past positions and use the resulting weights to mix their values. Past queries are not needed again: they already produced their own outputs. Therefore the cache retains past keys and values, while the new token computes its own $q_t$, $k_t$, and $v_t$.

For one head, the decode operation resembles

where cached matrices grow when $k_t$ and $v_t$ are appended. This happens at every cached layer. The cache is a consequence of exact causal attention, not an approximation to it.

Ignoring allocator overhead, an intuitive memory model is

where the factor two represents keys and values, $L$ is layer count, $T$ cached positions, $H_{kv}$ KV heads, $D$ head dimension, and $B$ bytes per element. Multiply again by active sequences. This formula explains why long context can exhaust memory even though model weights remain unchanged.

Multi-head attention uses separate K/V heads alongside query heads. Multi-query and grouped-query attention share K/V across query heads, reducing $H_{kv}$ and therefore cache traffic and capacity. Quantized cache formats reduce $B$, with accuracy and kernel-support trade-offs. Sliding windows or eviction reduce effective $T$, but then old tokens are no longer available to ordinary attention.

Cache layout matters to performance. Decode reads a long history to produce a small amount of new computation, so moving K/V data can dominate. Tensors need layouts that kernels consume efficiently, while a server needs flexible allocation for sequences that grow unpredictably. Reserving every request’s maximum length wastes memory; repeatedly moving contiguous buffers creates fragmentation and copies. Paged allocation addresses this ownership problem.

Prefix caching shares already computed blocks when requests begin with exactly the same token prefix, such as a common system prompt. Safe sharing requires the same model, adapter, positional treatment, and token sequence. Text that looks identical after rendering may tokenize differently. Cache keys must never cross authorization boundaries when the cached representation or timing can reveal private context.

Beam search and speculative decoding complicate ownership. Several candidates may share a prefix, branch, and discard paths. Copy-on-write block tables avoid copying the common prefix. Cancellation must decrement references exactly once. A leak gradually removes serving capacity; premature reuse corrupts one request with another’s state.

The cache does not make decode constant-time in context length. Projection of the old prefix disappears, but the new query still reads and attends over retained K/V states unless the architecture uses a bounded window or different mechanism. The durable model is a transcript: computation already performed becomes memory, saving arithmetic at the price of growing bandwidth, capacity, lifecycle, and privacy responsibilities.