TensorRT-LLM

01 · Concept

Concept

Your vLLM deployment works, and you want another NVIDIA-focused stack. Current documentation changes the premise: in August 2026 TensorRT-LLM removed its serialized TensorRT-engine backend. PyTorch is now the sole execution backend, with AutoDeploy remaining available on top of it.

There is no separate conversion or trtllm-build step. The current command is trtllm-serve pointed directly at a Hugging Face checkpoint. This is not eager PyTorch: the runtime still supplies NVIDIA-specialized kernels, in-flight batching, paged KV, quantization, speculative decoding, and parallelism. Pin the checkpoint and runtime, configure memory and scheduling, and benchmark the real workload; do not maintain an engine matrix by frozen shape.

Quantization can still release substantial serving capacity by shrinking the resident weights and, when the runtime supports a validated low-precision KV format, the cache as well. Do not infer a multiplier from bit width alone: lesson 7.2 owns the model-specific KV derivation, and lesson 9.3 owns the full accelerator budget, including runtime reserve and recurrent state. Recompute there with the exact artifact and runtime configuration, then require the correctness canary from lesson 8.1 before treating the recovered capacity as usable.

Serving is otherwise familiar. In-flight batching is TensorRT-LLM’s name for the iteration-level scheduling of lesson 7.8: requests join and leave between decode steps rather than waiting for a fixed batch to drain. Paged KV, chunked prefill, and speculative decoding all have counterparts here. The server and configuration surface differ from vLLM, but the concepts transfer intact, which is why this lesson can spend its space on the parts that do not.

The operational bill is now a compatibility and configuration matrix, not a serialized-engine matrix. A runtime update can change kernels, memory use, or operator coverage; a new precision can change quality; a new traffic profile can need different scheduling. Pin versions, preserve correctness canaries, record realized KV capacity, and revalidate after every material change.

Current TensorRT-LLM provides an NVIDIA-specialized runtime without a separate AOT engine. The price is NVIDIA ecosystem dependence and validation per version, precision, model, and workload. The right choice passes the same canaries and delivers more goodput inside the SLO.