Serving stacks: vLLM, SGLang, TensorRT-LLM

01 · Concept

Concept

You have chosen Qwen3.8-27B; now something has to serve it. The model card names vLLM, SGLang, and Transformers as supported paths, SGLang maintains an official cookbook page for this exact model, and TensorRT-LLM is the NVIDIA-maximal option. This lesson is deliberately a decision map, not a manual: each stack gets its own lesson in Track 8, where the flags, configs, and failure modes live. Here you learn only which door to walk through.

The first gate is arithmetic, not software. The weights cost 54 GB in bf16, and lesson 9.3 owns what is left over on an 80 GB card: 23.3 GiB of KV pool once the framework’s own reservation is subtracted, which is roughly 381,700 tokens at lesson 7.2’s 64 KiB per token, or eleven 32K sessions. Do not reach for the obvious $80 - 54$ here — that subtraction is the worked mistake 9.3 opens with. It inflates the pool only slightly, but dividing it by context alone, as though a sequence cost nothing but its cache, overstates how many conversations fit by a fifth. If your hardware budget is a 24 or 48 GB card, this whole lesson is premature: you are on the quantized-artifact path through lesson 7.12 and the local runners of Track 8, not choosing among data-center engines.

With capacity settled, match the engine to the workload’s shape. vLLM is the default for a high-concurrency API over variable-length conversations on GPUs you control: it grew around exactly the PagedAttention and continuous-batching machinery of lesson 7.8, serves an OpenAI-style API, and covers the broadest open-model surface — lesson 8.4. SGLang earns its place when your traffic is not isolated prompts but structured programs: agent pipelines making many related calls, heavy shared prefixes, and schema-constrained JSON output; its RadixAttention prefix reuse and program-level scheduling target that structure, and its official Qwen3.8-27B cookbook makes it a first-class citizen here — lesson 8.5. TensorRT-LLM fits a committed NVIDIA fleet chasing the last percent of latency with FP8 engine builds, provided an ops team can own compilation, version pinning, and rebuilds when shapes change — lesson 8.6. Plain Transformers remains right for one researcher on one GPU doing correctness work, not throughput — lesson 8.2. No GPUs at all? A managed endpoint such as Workers AI serves this model per token — USD 0.45 per million input tokens and USD 3.20 per million output as of Aug 2026 — and lesson 8.11 does that cost math.

The classic wrong turn is picking by headline throughput. A vendor benchmark at one batch size, one prompt length, and one quantization tells you almost nothing about your goodput. The corrected order: first confirm the engine version supports your exact model — then verify correctness (tokenizer and chat-template parity, thinking-mode handling, stop behavior, structured output), and only then benchmark pinned versions on your hardware with your prompt and output distributions, identical decoding settings, and production-like concurrency. Measure time to first token, inter-token latency, and goodput against your latency objective — not a single peak number.

Operations often decide ties: metrics and traces, health checks, rolling upgrades, cancellation cleanup, admission control, multi-tenancy, container size, and cold-start time. A marginal kernel gain rarely justifies an engine your team cannot diagnose at 3 a.m. Mixing stacks is legitimate — throughput-oriented batch generation on one, interactive serving on another — but every extra runtime multiplies conformance testing.

Hold the map, not the manuals: hardware gate first, then workload shape — concurrency to vLLM, program structure to SGLang, NVIDIA-maximal latency to TensorRT-LLM, simplicity to Transformers, no-hardware to a managed endpoint — then correctness, then your own benchmark. Track 8 now descends the stack properly, and it starts where every engine secretly begins: lesson 8.1, PyTorch as the reference runtime.