SGLang

01 · Concept

Concept

Your traffic is not what lesson 8.4 assumed. Instead of independent users asking independent questions, you are running an agent: one task fans out into forty model calls, each of which begins with the same system prompt, the same tool schemas, and the same few-shot examples — about three thousand tokens of identical preamble — and diverges only in the last paragraph. Every call must return a JSON object your code will parse. On a general-purpose engine this workload is quietly pathological, and understanding why is the whole reason SGLang exists.

Price the waste with the number you already have. Lesson 7.2 established 64 KiB of KV per token for Qwen3.8-27B. A three-thousand-token preamble therefore occupies

of cache. Forty concurrent calls each holding their own copy come to roughly 7.3 GiB of duplicate KV — substantial pressure in any realized pool, whose complete budget belongs to lesson 9.3. The compute is just as bad: the preamble is prefilled forty times, one hundred and twenty thousand tokens of attention work to produce a prefix the engine already computed thirty-nine times that second.

RadixAttention is the fix, and its shape follows from what the cache actually is. KV blocks are a function of the token ids that produced them and their positions, so two sequences that begin with the identical token run have identical blocks for that run. SGLang therefore organizes the block pool as a radix tree keyed on token ids: an arriving request walks the tree from the root, reuses every block along the longest matching path, and allocates only for the suffix where it diverges. Eviction is least-recently-used over that tree, so hot prefixes — your system prompt — stay resident while one-off conversations age out. In our example the preamble is stored once at about 188 MiB and prefilled once, recovering roughly seven gibibytes of pool and thirty-nine fortieths of the preamble’s prefill cost.

The second thing SGLang treats as first-class is that your forty calls are a program, not forty unrelated requests. They have dependencies, they fork and join, some of them are speculative branches you will discard. When the runtime knows the program structure it can schedule with that knowledge — keeping a branch’s prefix pinned while its children run, batching the siblings, ordering work so that a shared parent is computed before the fork rather than forty times after it. An engine that only sees independent HTTP requests can recover some of this after the fact through prefix matching, but it cannot anticipate it.

Structured output is the third piece, and it is what makes the JSON requirement a guarantee rather than a hope. Lesson 7.4 described sampling as drawing from the distribution over the vocabulary. Constrained decoding compiles your JSON schema or grammar into a state machine, and at every step masks the logits of every token that cannot legally follow the text so far before sampling happens. The model cannot emit a stray prose apology in front of the brace, cannot close an array it did not open, and cannot invent a field name outside the schema — not because it was asked nicely, but because those tokens had their probability set to zero. Two practical notes: compiling a grammar costs real time, so schemas should be reused and cached rather than constructed per request; and constraint applies only to shape.

The hybrid architecture adds a wrinkle to prefix sharing that is worth stating precisely, because it is the deepest thing in this lesson. Only the 16 full-attention layers store per-token KV blocks, which are exactly what a radix tree can share. The 48 Gated DeltaNet layers hold a fixed-size recurrent state that is a summary of everything processed so far — it has no per-token structure and cannot be indexed into or partially reused. For a request to resume from a shared prefix, the engine must therefore also have checkpointed the recurrent state at that prefix boundary and be able to restore it. The reference implementation stores about 144 MiB per sequence when that state is fp32 — the same order as the 188 MiB of shared KV in this example, not a negligible add-on. A serving runtime may use another state dtype or layout, so inspect its realized allocation. This is a genuinely different mechanism from block sharing, and whether a given release implements it — rather than silently falling back to full recomputation, or worse, reusing blocks against a stale state — is a per-version question to verify, not to assume. SGLang publishes an official cookbook page for this exact model, which is the right starting point precisely because it pins the versions and settings the maintainers have exercised.

One boundary rule survives all of this unchanged. Reuse is exact-match on token ids, so no content crosses between requests that did not already share it verbatim — but a cache hit is faster than a miss, and timing is observable. Treat cross-tenant prefix sharing as a decision to make deliberately, keep tenants in separate pools when the prompt itself is confidential, and re-read lesson 7.2’s warning about cached keys and values being a rich record of what a user said.

The map from lesson 7.11 said SGLang earns its place when your traffic is programs rather than prompts. Now you can say why in numbers: seven gibibytes of pool and thirty-nine redundant prefills recovered from one workload, and a JSON contract enforced at the sampler instead of hoped for in the prompt.