KV cache sizing and memory budgets

01 · Concept

Concept

You have one 80 GB accelerator and a product requirement: serve Qwen3.8-27B to as many simultaneous conversations as the card allows, each with an 8,192-token window. Somebody subtracts 54 for the weights from 80, divides the remainder by lesson 7.2’s 64 KiB per token, and reports 48 users. The worked result is 36: the naive estimate is high by one third, and the failure mode is an out-of-memory crash under load. This lesson does the budget line by line because everything downstream prices against it.

Start with usable capacity rather than brochure capacity. A part marketed as 80 GB carries 80 GiB-class HBM and reports roughly 79.6 GiB to the runtime once ECC and reserved firmware regions are taken out, so the number you can actually allocate is smaller than the number on the box. Doing the entire budget in one unit family is the only defence, so convert everything to gibibytes and stay there. The bf16 weights, about 54 GB from the canonical figure, are 50.3 GiB. Next comes a line the naive subtraction omits entirely: the runtime reserve. The driver context, activation buffers for the largest batch you intend to run, temporary workspace for the attention and feed-forward kernels, and allocator fragmentation together consume several gibibytes before a single token of cache exists. Six gibibytes is an illustrative and not especially generous figure; serving frameworks encode the same idea as a utilization fraction, defaulting to holding back roughly a tenth of the device. The remainder is the pool:

Now price one sequence, and price it completely. Lesson 7.2 derived the cache at 4 KiB per token per caching layer, 64 KiB per token across the 16 full-attention layers, and that derivation is not repeated here. An 8,192-token conversation therefore holds

of keys and values. But the 48 Gated DeltaNet layers are not free either. In the reference Transformers path, each keeps a 3 MiB float32 matrix state, 144 MiB across the layers, excluding the small convolution state. The state belongs to the sequence, not the model. Every concurrent conversation carries its own copy. The true per-sequence cost is 512 plus 144, or 656 MiB, and the pool holds

Thirty-six, not forty-eight. The three corrections that closed the gap were the unit conversion, the runtime reserve, and the constant recurrent state.

Push the same pool to the other extreme. A full 262,144-token sequence holds 16,384 MiB of K/V plus 144 MiB of matrix state, so the pool holds one. At 32,768 tokens each sequence costs 2,048 plus 144, or 2,192 MiB, and the card serves ten. Concurrency is a function of the context length you promise.

Redo the budget with FP8 weights and an FP8 cache. The weights halve to about 27 GB, or 25.15 GiB, and the cache halves to 32 KiB per token. The pool becomes about 48.45 GiB or 49,613 MiB. Each 8,192-token sequence now costs 256 MiB of cache plus the unchanged 144 MiB reference state, 400 MiB in total, so the card serves

and five full-native-context conversations rather than one. That is about a 3.4-fold gain in concurrency from this paired precision change.

Omitting the DeltaNet state predicts 46 bf16 sequences instead of 36, about 28 percent too many, and 193 FP8 sequences instead of 124, about 56 percent too many. The state is constant, so as each variable term shrinks its relative weight grows; many copies of a fixed 144 MiB become a major budget line.

Carry two habits out of this lesson. Budget in one unit family, and price a sequence completely rather than pricing only its cache. Lesson 9.4 changes the capacity line by changing the silicon, and lesson 9.10 turns these occupancy figures into cost per token.