Serving stacks: vLLM, SGLang, TensorRT-LLM

1

Concept

A serving engine is more than a fast matrix-multiplication library. It loads checkpoints, chooses kernels, shards weights, allocates KV state, batches changing requests, applies decoding rules, exposes APIs, streams tokens, records metrics, and recovers from failure. vLLM, SGLang, and TensorRT-LLM cover overlapping pieces with different design centers.

vLLM grew around PagedAttention and iteration-level scheduling. Its block-based KV management and continuous batching target high utilization under variable-length requests. It supports familiar completion-style APIs and a broad open model ecosystem. Features such as tensor parallelism, prefix caching, adapters, quantization, and speculative decoding depend on the specific release and model architecture, so a feature matrix must be checked against the pinned version.

SGLang combines a frontend language for structured model programs with a serving runtime. Applications often execute repeated prompts, branching generation, tool-like phases, or multi-call workflows with shared prefixes. Representing those structures explicitly creates opportunities for prefix reuse and scheduling across the program. SGLang can also serve conventional APIs; its distinctive idea is that inference programs contain structure beyond isolated prompt strings.

TensorRT-LLM is an NVIDIA-centered stack with optimized kernels, graph or engine building, quantization paths, in-flight batching, and multi-GPU execution. Tight integration can unlock hardware-specific features, but deployment requires matching supported GPUs, CUDA and driver stacks, model definitions, plugins, and build configuration. An engine optimized for one shape or feature set may need rebuilding when assumptions change.

These summaries age quickly. Compare capabilities from official documentation and the exact versions being deployed. “Supports model X” may exclude a new architecture variant, adapter mode, cache dtype, constrained decoder, or quantization format. A silent fallback can be correct but slow; an unsupported combination should fail clearly rather than produce subtly wrong outputs.

Selection starts with correctness. Verify tokenizer and chat-template parity, logits or deterministic outputs on controlled prompts, stop behavior, tool schemas, structured constraints, and long-context positions. Then test the production workload: distributions of prompt and output lengths, concurrency, streaming clients, cancellation, prefix repetition, and adapters.

Performance needs several axes. TTFT rewards fast admission and prefill. Inter-token latency matters for interactive reading. Request throughput matters for capacity. Goodput counts only requests meeting latency objectives. Memory headroom determines burst tolerance. Benchmarking one batch size or quoting vendor maxima cannot answer all of these.

Operations often decide the winner. Examine metrics, traces, request IDs, health checks, rolling upgrades, model load time, cancellation cleanup, backpressure, admission control, multi-tenancy, and failure isolation. Container size and cold start matter in autoscaled environments. A marginal kernel gain may not justify an engine the team cannot diagnose.

Architecture can mix stacks. Offline batch generation may value throughput, while interactive serving values tail latency. A router can direct supported models to specialized engines, but every extra runtime expands compatibility testing and observability. Standardize the application contract while retaining engine-specific conformance tests.

The durable selection rule is empirical and workload-specific. First establish exact feature and correctness support. Then benchmark pinned versions on target hardware with representative load and identical decoding. Finally evaluate operational ownership. The best serving stack is not the one with the loudest peak number; it is the one that meets the system’s quality and latency contract predictably.