Prefill vs decode: two different machines

1

Concept

Inference is often described as one forward pass repeated many times, but the first pass and later passes have very different shapes. Prefill processes the input prompt and creates attention state. Decode consumes that state to generate new tokens one position at a time. Treating them as the same workload hides the reason optimizations help one metric and hurt another.

During prefill, the model receives many prompt positions together. Linear layers multiply weight matrices by activation matrices with a substantial token dimension. Attention computes interactions among allowed prompt positions, and each layer writes keys and values into the cache. Accelerators can exploit parallelism across tokens, heads, and batch entries. Long prompts create considerable work, but that work can reach high arithmetic utilization.

The output of prefill is the logits for the first generated token plus a populated KV cache. Therefore prompt length, queue delay, kernel efficiency, and prefix-cache hits strongly influence time to first token (TTFT). A service optimized only for total tokens per second can batch prefills for too long and make an interactive user wait.

Decode has only one new position per active sequence at each step. The model still traverses every layer and reads its weights, but the activation matrix is thin. It also reads cached keys and values for attention. There may be too little arithmetic per byte moved to saturate compute units. This is why decode is commonly described as memory-bandwidth-bound, although the actual bottleneck depends on batch size, model, cache length, kernels, and hardware.

A roofline model clarifies the distinction. Hardware has a ceiling for arithmetic throughput and another for memory bandwidth. A kernel’s arithmetic intensity is operations performed per byte transferred. Low-intensity work hits the bandwidth roof first; high-intensity work may approach the compute roof. Prefill generally exposes higher intensity than single-sequence decode. Batching decode combines independent sequences and reuses weights across them, increasing useful work per weight read.

Batching is not free. Waiting to form a larger batch raises queue latency. Active sequences finish at different times and have different cache lengths. Continuous batching inserts new requests as old ones finish, trading a static rectangular batch for a changing set of token steps. The scheduler must balance throughput, TTFT, inter-token latency, fairness, and memory.

Prefill and decode can interfere. A large prefill consumes compute and memory bandwidth while decode requests need regular token delivery. Chunked prefill divides a long prompt so decode steps can run between chunks. Some systems disaggregate the phases onto different workers, but then KV state must move across a network or interconnect. The saved interference must exceed transfer and coordination costs.

Metrics should be phase-aware. Record queue time, prefill duration and prompt throughput separately from decode step time and output throughput. Plot percentiles by prompt length, output length, and concurrency. Aggregate averages can hide a scheduler that serves bulk jobs efficiently while starving interactive traffic.

Optimization follows the distinction. Faster attention kernels and prefix reuse target prompt work. Weight quantization, cache layout, speculative decoding, and larger decode batches target repeated generation costs. Scheduling connects both. The durable mental model is not one machine doing one task, but a parallel prompt processor handing state to a latency-sensitive token engine.