vLLM

01 · Concept

Concept

You have one 80 GB GPU, a copy of Qwen3.8-27B, and a request to stand up an internal chat API for perhaps forty engineers. Two numbers decide whether that works: how much of the card the engine is allowed to claim, and how long a single conversation may become. Get them wrong in one direction and the server refuses to start; wrong in the other and it dies under the first real burst of traffic. This lesson shows the operational sizing loop, because everything else about vLLM — the OpenAI-shaped API, the scheduler, the metrics — is documentation you can read, and mistaking a configured ceiling for realized capacity is the part people get wrong.

vLLM exists because of lesson 7.8. Its scheduler admits and retires requests at iteration boundaries rather than holding a fixed batch until the slowest member finishes, and its cache is paged: fixed-size blocks handed out as tokens are produced, so a session that stops after four thousand tokens never paid for the thirty-two thousand it was permitted. The engine’s job at startup is to decide how large that block pool should be, and it does so by subtraction rather than by asking you.

Treat --gpu-memory-utilization as a ceiling, not as the KV pool. The engine pays resident weights, activation peaks, CUDA graph pools, allocator fragmentation, and architecture-specific state before it exposes any cache capacity. Lesson 9.3 owns that full accelerator budget, while lesson 7.2 owns this model’s cache and recurrent-state derivation. In the reference implementation the 48 Gated DeltaNet layers keep about 144 MiB of recurrent state per sequence when that state is fp32; another runtime can choose a different dtype or layout, so copying that byte count into a vLLM capacity claim without checking the implementation would be just as wrong as ignoring the state.

The source of truth is therefore the launch you will serve. Pin the checkpoint and vLLM release, leave deliberate headroom, and read the startup lines reporting the KV pool actually allocated and the maximum concurrency implied by your chosen context length. --max-model-len must fit at least one request inside that realized pool. If it does not, the engine should reject the configuration at startup; lower the limit, change a validated memory format, or add capacity, then measure again.

Paging changes admission after that safety condition is met. Blocks are allocated for tokens actually present rather than for every token a request is allowed to reach, so ordinary shorter conversations can coexist at much higher concurrency than a worst-case maximum-length division suggests. Size the hard limit from realized allocation, then size traffic from the measured distribution of prompt and completion lengths.

vllm serve Qwen/Qwen3.8-27B \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Do not trust the estimate above over the engine. vLLM prints the size of the KV pool it actually allocated and the maximum concurrency that implies at your chosen context length. Read that line, compare it with your arithmetic, and reconcile any large gap before you benchmark anything — a mismatch usually means the workspace reserve or the recurrent-state accounting is different from what you assumed.

If the realized pool cannot admit the full native context, no flag can turn that failed allocation into usable capacity. The honest options are more memory, sharding across GPUs with tensor parallelism (lesson 5.8 for the mechanics, lesson 9.9 for the interconnect cost), or a runtime-supported lower-precision memory format that has passed the correctness canary from lesson 8.1. Measure the new realized pool rather than projecting a multiplier from bit width.

Carry the measurement chain, not a memorized set of flags. The utilization fraction bounds the engine, the startup log reports the pool it actually realized, and --max-model-len bounds the largest single claim against that pool. Lessons 7.2 and 9.3 own the arithmetic behind those quantities; this lesson owns how vLLM exposes and enforces them.