Prefill vs decode: two different machines

01 · Concept

Concept

Serve Qwen3.8-27B and watch the timings: an 8,192-token prompt is absorbed in about the time it takes to read this clause, then the answer trickles out token by token. Same weights, same GPU — yet the two phases run at speeds separated by orders of magnitude. They are effectively two different machines. Prefill processes the prompt and creates state; decode consumes that state one token at a time.

During prefill, every prompt position moves through the network together. Linear layers multiply weights by activation matrices with a token dimension in the thousands; the 16 attention layers compute interactions among prompt positions and write keys and values, while the 48 Gated DeltaNet layers fold the prompt into their recurrent states. Accelerators love this shape — parallelism across tokens, heads, and batch entries — and long prompts can approach the hardware’s arithmetic ceiling. Prefill’s output is the first token’s logits plus populated state, which is why prompt length and kernel efficiency dominate time to first token.

Decode presents the opposite shape: one new position per active sequence per step, through every layer.

Here is the wrong turn most people take when decode feels slow: assume the GPU is out of arithmetic and reach for more FLOPs. Run the numbers instead. For a weight matrix of $P$ parameters in bf16 (2 bytes each), processing $n$ tokens costs about $2nP$ operations against at least $2P$ bytes of weight traffic, so the arithmetic intensity is

Prefill with $n = 8{,}192$ prompt positions performs roughly 8,192 useful operations for every weight byte it fetches. Batch-1 decode has $n = 1$: about one operation per byte. And the bytes are substantial — Qwen3.8-27B’s bf16 weights total about 54 GB, essentially all of which must stream from memory on every single decode step. One operation per byte is far below what any modern accelerator needs to keep its arithmetic units busy, so the step is limited by memory bandwidth, not compute. Adding FLOPs would change nothing; the roofline model (Williams et al.) makes this precise, and lesson 9.1 turns the 54 GB-per-token weight read into an explicit tokens-per-second ceiling on H100-class memory bandwidth. The correction, not the naive guess, explains why decode remedies are all about bytes: quantized weights, batching to reuse each weight read across sequences, speculative decoding to verify several tokens per read.

The second stream of decode traffic is state, and here Qwen3.8-27B’s hybrid layout changes the classic picture. In a uniform attention model, every layer reads a KV history that grows with context, so decode gets progressively more expensive per token. In this model only the 16 full-attention layers do that, reading a cache that grows at the 64 KiB per token derived in lesson 7.2 — illustratively, at 100,000 cached tokens those layers stream about 6 GiB of K/V per step. The 48 DeltaNet layers instead read and update fixed-size matrix states totaling 144 MiB in the reference float32 path, the same at token ten and token 200,000. Three-quarters of the depth contributes constant decode traffic, so cost grows with context far more slowly than layer count alone would suggest — that flattening is precisely what the architecture bought.

Batching reshapes both regimes. Independent sequences decoded together share each weight read, multiplying useful work per byte; that is why $n$ in the intensity formula is really tokens-in-flight, not just prompt length. But batches are not free: waiting to form one raises queue latency, sequences finish at different times, and continuous batching (lesson 7.8) trades a static rectangle for a shifting set of token steps the scheduler must keep fair.

The phases also interfere. A large prefill monopolizes compute and bandwidth exactly while decode requests need their steady cadence; chunked prefill interleaves prompt pieces between decode steps, and some deployments disaggregate the phases onto separate workers, paying KV transfer costs to avoid the contention.

Metrics should respect the split: record queue time, prefill duration, and prompt throughput separately from decode step time and output throughput, in percentiles by prompt length and concurrency. Aggregate averages happily hide a scheduler that feeds bulk jobs while starving interactive users.

The durable mental model: a parallel prompt processor hands state to a latency-sensitive token engine that mostly moves memory. Everything in the rest of this track — IO-aware attention, paged caches, quantization, speculative decoding — is an attack on one side or the other of that boundary.