Choosing your stack

01 · Concept

Concept

You have met eleven ways to run the same model. Somebody now asks you which one to use, and the honest first move is to refuse the question as posed. There is no fastest stack, because speed is not a property of a stack — it is a property of the pairing of a stack with a model, a piece of hardware, and a traffic pattern. Change any of the three and the ranking changes. What you can have instead is a procedure that reaches a defensible answer in three questions, and a standard for what counts as evidence.

The first question is hardware, because it is the only one that can end the conversation immediately. Qwen3.8-27B in bf16 is 54 GB of weights before a single token of cache, which means an 80 GB-class accelerator, a multi-GPU shard, or a very large unified-memory machine. The pinned Q4_K_M text artifact is 17.1 GB (15.93 GiB), which opens a 24 GB card, a 32 GB Mac, and a well-specified laptop after runtime headroom is validated. More aggressive supported quants can be smaller; choose a smaller model only when no quant with acceptable quality, support, and latency fits. If your memory cannot hold weights plus the cache your context length demands, no engine choice rescues you.

The second question is workload shape, and it has more dimensions than people expect. Is traffic one person at a keyboard, or hundreds of concurrent sessions? Do requests share a long identical prefix — a system prompt, a policy document, a few-shot preamble — or is every prompt unique? Is the context two thousand tokens or two hundred thousand? Must output conform to a schema? Are images involved? Is thinking mode on, so generations are long? Each of these pushes toward a different engine, and they can conflict, which is why they are answered before a stack is named rather than after.

The third question is ops capacity, and it is the one engineers systematically under-weight. Somebody upgrades drivers. Somebody notices the p99 latency drifting. Somebody is awake when a kernel regression lands. A stack that is ten percent faster and consumes a quarter of an engineer is not ten percent better.

Answer the three and a map falls out. One machine, one user, Apple silicon: MLX from lesson 8.9, or a llama.cpp runner from lesson 8.8 if you want the convenience and the registry. One NVIDIA GPU serving an internal API: vLLM from lesson 8.4, which is the default and deserves to be. Many concurrent users with heavily shared prefixes, or output that must be valid JSON: SGLang from lesson 8.5, whose prefix-sharing and constrained decoding are exactly those two problems solved. An NVIDIA-only fleet, throughput as the dominant cost, and engineering time to spend on runtime and compatibility validation: TensorRT-LLM from lesson 8.6. A heterogeneous or future-heterogeneous fleet, with tolerance for a young ecosystem: MAX from lesson 8.10. Low, spiky, or unpredictable volume with no ops rotation: the managed endpoint from lesson 8.11.

Work one case fully. A team of three engineers serves 200 daily users, mostly short chats at around 8,192 tokens of context, and needs replies as validated JSON. They can fund one 80 GB accelerator. Step one, hardware: lesson 9.3 does this budget line by line and lands on a 23.3 GiB cache pool once weights, workspace and activations are taken out. Using the complete sizing from lesson 9.3, with lesson 7.2’s KV-cache rule and the reference Transformers path keeping Gated DeltaNet state in fp32, roughly 36 live 8K sequences fit concurrently — far above 200 users spread across a working day. One card is enough, and they do not need to quantize. Step two, workload: the JSON requirement and a shared system prompt point squarely at SGLang; identical prefixes across users mean the prefill for that prompt is computed once and reused, which is throughput they get for free rather than by tuning. Step three, ops: three engineers building a product is thin cover for a serving deployment, so they compare against the managed option using lesson 8.11’s arithmetic before committing — and if their token volume resembles that lesson’s example, the metered bill is small enough that owning a card would be buying idle silicon.

Notice what decided it. Not a benchmark. Capacity arithmetic, a feature requirement, and a headcount.

So what makes a comparison fair? Six things must be identical across arms: the model revision, the numeric precision of both weights and KV cache, the decoding configuration — and for this model that means one of the two published presets, since thinking mode at temperature 1.0 generates far more tokens than the instruct preset at 0.7 and will move throughput on its own — the input and output length distributions, the concurrency level and arrival pattern, and the measurement window with a stated warm-up policy. Then report the right things: throughput alongside time-to-first-token and p50 and p99 end-to-end latency, because a stack that wins on aggregate tokens per second while doubling tail latency has lost for an interactive product. And state whether prefix caching was warm, since a benchmark that replays one prompt measures a cache, not a model.

Write the run as a four-part benchmark passport:

1. Identity: exact model and artifact revision, runtime/build, driver and kernels, template, weight/KV/recurrent-state formats, and speculative decoder.
2. Protocol: prompt or benchmark version, harness and scorer revision, sampling and reasoning policy, output budget, input/output distributions, concurrency and arrival pattern, prefix-cache state, warm-up, repetitions, and random seeds where applicable.
3. Validity: correctness canaries before timing, configured versus realized capacity, preregistered discard rules, and separate counts for admission rejection, length truncation, schema/tool failure, and completed-but-wrong output.
4. Results: TTFT, inter-token or decode rate, end-to-end p50/p95/p99, aggregate throughput, peak memory, errors and restarts, plus wall power or energy only when it was actually measured. Preserve raw requests, responses, and logs whenever licensing and privacy permit.

This structure extends the article’s deployment lesson with the same principles used by MLPerf and HELM: define the scenario, standardize the conditions, validate accuracy, expose multiple metrics, and retain enough raw evidence to audit the conclusion. A passport does not make unlike workloads comparable; it makes the boundary visible.

A trustworthy benchmark also needs a complete identity. In the public Qwen3.8-27B campaign, the same RTX 5090 target moved from about 71.8 tok/s on an SGLang NONE profile to 151.27 tok/s on a patched TurboQuant 4-bit KV + MTP3 stack, but the faster number was accepted only after exact-output, JSON, tool, coding, and multi-turn canaries passed; the stock accelerated launch was a correctness HOLD. Context claims needed the same discipline: one profile was configured for 262,144 tokens but realized only 42,624 KV tokens, while a different long-context profile recovered three needles at 261,212 input tokens with a much longer TTFT. And two quality runs were discarded because the controller OOMed and then the response cache became read-only. Those are not footnotes. They are part of the result.

The durable posture is to decide with arithmetic and confirm with measurement, in that order. The memory budget tells you what is possible, the workload tells you what is appropriate, headcount tells you what is sustainable, and a controlled experiment on your own traffic tells you whether you were right — which is a discipline no vendor page can hand you.