Hybrid layouts: three DeltaNet, one attention

01 · Concept

Concept

You now hold two mixers and a design problem. Full attention exposes per-token keys and values across the context: a query can directly address any past position, although this does not guarantee verbatim recovery, at the price of storing keys and values for the entire context and re-reading them every step. Gated DeltaNet is cheap: constant-size state, constant per-token cost at any length, at the price of lossy learned compression — a fixed board cannot promise perfect recall of an arbitrary clipping from half a million tokens ago. A model that wants a 262,144-token native context must decide, layer by layer, which price to pay. Qwen3.8-27B’s answer is a hybrid layout: a four-layer motif — Gated DeltaNet, DeltaNet, DeltaNet, full attention — repeated sixteen times. That is 48 DeltaNet layers, 16 attention layers, one archivist every fourth floor.

Work the memory consequences step by step, because the intuitive reading gets them backwards. The reflex says: forty-eight recurrent layers, each dragging a state matrix from token to token — surely that is where long-context memory goes, with the sixteen attention layers a rounding error. Now check each type against its actual contract. A DeltaNet layer’s state is a fixed set of matrices, edited in place; token 200,000 costs exactly what token 200 cost, and the layer stores nothing per token. An attention layer is the opposite: its direct per-position access relies on storage — every generated token appends keys and values that all future queries may consult, so its memory grows linearly with context, multiplied across all sixteen such layers. At the full native window the totals are not close: the sixteen attention layers’ key-value cache runs to roughly 16 GiB per sequence, while all forty-eight DeltaNet states together occupy about 144 MiB when the reference implementation keeps recurrent state in fp32, constant in length; another runtime dtype would change the byte count — a contrast of more than two orders of magnitude, in the opposite direction from the reflex. Lesson 7.2 derives both numbers from the head counts and dimensions; here the shape of the result is what matters: the minority layer type dominates the memory bill precisely because it is the type that remembers everything.

That asymmetry is the whole argument for the ratio. If every layer were full attention, the growing cache would be four times larger and long contexts would price themselves out of memory. If every layer were DeltaNet, the model would face needle-in-a-haystack retrieval — quote the exact serial number from page two of a four-hundred-page document — armed only with lossy summaries. The 3

What makes the mixture function as one model rather than two interleaved ones is the block contract from lesson 4.11. Both layer types read the same 5120-wide residual stream through the same pre-norm RMSNorm and add back updates of the same shape. A retrieval performed by the attention layer on floor 3 — a representation routed from a distant token by content-addressed attention — is written into the stream, where the DeltaNet layers on floors 4 through 6 can transform, route, and build on it with their cheap machinery. The archivist retrieves; the editors process. Division of labor, coordinated through the shared board, with the identical gated FFN doing the heavy feature transformation on every floor of both kinds.

The layout also shapes where different capabilities can live. Cross-context operations that benefit from direct per-position access — copying a long identifier, matching a bracket opened 100,000 tokens ago, induction over distant repetitions — must route through one of the sixteen attention layers, because only they can address arbitrary positions without fixed-state compression. Recency-weighted processing, local composition, and running-state accumulation sit naturally in the DeltaNet majority, whose kernel-4 convolution and gated state handle short- and medium-range structure without touching a growing store. Interpretability work on hybrid models starts from this map: if a behavior needs direct long-range access, you know which sixteen layers to search.

Qwen did not invent the pattern — 2025–26 produced a family of such designs, interleaving state-space or linear-attention blocks with full attention in Jamba-style and Granite-H lineages — but a 3

Where this leads is inference. The hybrid layout is, at bottom, a bet about serving economics: that a model can keep attention’s retrieval quality while shrinking the memory that grows per token to a quarter of the layers — and that this determines what contexts, batch sizes, and hardware are feasible. Pricing that bet requires deriving the cache arithmetic properly: bytes per token per layer, per sequence, per gigabyte of accelerator memory. That derivation — and the full 16 GiB versus approximately 144 MiB reference-fp32 accounting this lesson only gestured at — is lesson 7.2, where the KV cache gets its own treatment.