PagedAttention & continuous batching

01 · Concept

Concept

You have one 80 GB accelerator and you want to serve Qwen3.8-27B to as many concurrent users as possible. The weights occupy 54 GB in bf16 (lesson 9.3 walks the full budget), and lesson 7.2 established the model’s KV bill: only the 16 full-attention layers cache keys and values, at 64 KiB per token, reaching 16 GiB per sequence at the full 262,144-token native context. The serving question is where that growing, unpredictable state should live.

The classic wrong turn is contiguous reservation at the maximum. A request could in principle grow to native context, so reserve 16 GiB up front. Lesson 9.3 owns the complete device budget: after unit conversion, weights, and runtime reserve, its bf16 example leaves a 23.3 GiB state pool. One maximum reservation fits. Your 80 GB GPU serves exactly one user, and since a typical chat uses a few thousand tokens, almost all of that reservation sits empty. Reserving the current length instead forces buffers to grow, and growth means relocation, copies, and fragmentation across dozens of differently sized allocations.

PagedAttention, introduced with vLLM in 2023, borrows the operating-system idea of virtual memory. A sequence’s logical KV positions are divided into fixed-size blocks — say 16 tokens, which at Qwen’s 64 KiB per token makes a tidy 1 MiB physical block. Physical blocks live anywhere in the cache pool; a per-sequence block table maps logical order to physical addresses, and the attention kernel follows the mapping while reading keys and values. When a sequence outgrows its last block, the allocator hands it another free block without moving anything. Waste is bounded to the unused tail of one block per sequence, not an entire maximum reservation.

Now redo the capacity example with paging. Suppose your users run long-document sessions around 32,768 tokens of context. Per sequence, KV costs $32{,}768 \times 64\ \text{KiB} = 2\ \text{GiB}$. Add the 144 MiB reference float32 DeltaNet matrix state from lesson 7.2 and each sequence costs 2,192 MiB. The 23,859 MiB pool from lesson 9.3 therefore holds ten such sessions, with that lesson’s runtime reserve already removed. Against the single user of the reservation scheme, that is an order of magnitude from allocation policy alone.

The hybrid architecture adds a wrinkle worth naming. Paging governs the 16 attention layers’ growing KV; the 48 DeltaNet layers’ state is fixed-size and per-sequence, so a hybrid-aware server manages two pools with different lifecycles. How vLLM handles this concretely is lesson 8.4’s territory.

Indirection also enables sharing. Parallel samples, beam candidates, or requests with an identical prompt prefix can point at the same read-only blocks; when a branch diverges, copy-on-write allocates a fresh block for the changed positions, and reference counts decide when a shared block returns to the pool.

Allocation solves where state lives; continuous batching solves when each request runs. A static batch groups requests and runs until every member finishes, so one long response leaves the other lanes idle. Continuous batching revisits membership at every iteration boundary: finished or canceled sequences leave, waiting sequences enter, active ones advance one token. The scheduler juggles free blocks, maximum batch tokens, priorities, and the prefill/decode mix — often chunking long prefills so a new arrival’s 30,000-token prompt does not stall everyone else’s next token. Because capacity is visible as countable blocks rather than vague free memory, admission control can reason in blocks and reject early with a clear error instead of failing deep into generation.

Watch the operational metrics that make this machinery legible: free and used blocks, allocation failures, tail waste, prefix hit rate, preemptions, queue delay, time to first token, and inter-token latency, sliced by request class so throughput does not hide starvation.

The durable mental model has two layers. Paged allocation turns a growing, irregular cache into fixed blocks plus a logical directory, so concurrency is set by real usage rather than pessimistic reservations. Continuous batching turns the batch into a living population that refills at every step. Together they are why the difference between one and ten concurrent Qwen sessions on the same silicon is software.