Gated DeltaNet: the mixer in 48 of 64 layers

01 · Concept

Concept

Attention answers every question by re-reading the entire past: each new token’s query scans all stored keys, and the store grows without bound. Lesson 3.6 showed the alternative — linear attention as a recurrent network whose state is a single matrix of fast weights, written by outer products and read by queries — and also showed its flaw: a memory you can only add to is a memory that interferes with itself. Qwen3.8-27B bets most of its depth on the repaired version. Forty-eight of its sixty-four layers mix tokens not with attention but with Gated DeltaNet, and this lesson opens that mixer.

Start from the failure, concretely. Additive fast weights maintain $S_t = S_{t-1} + v_t k_t^{\top}$ and answer queries with $S_t q_t$. Store “the capital is Paris” under key $k$, and later — the government moved — store “the capital is Brasília” under the same key. Step through the naive update: after the first write, $S k = v_{\text{Paris}}$ (take $k$ unit-norm). After the second, $S k = v_{\text{Paris}} + v_{\text{Brasília}}$. The memory now retrieves a superposition of a stale fact and a fresh one, and every further reuse of the key deepens the pile. Nothing in the additive rule can ever remove information.

The delta rule fixes the update by reading before writing. First ask the memory what it currently returns for the incoming key: $\hat v = S_{t-1}k_t$. Then write only the error, scaled by a data-dependent write strength $\beta_t \in (0,1)$:

Rerun the capital example with $\beta = 1$: the $(I - k k^{\top})$ factor erases exactly what $k$ retrieved — $v_{\text{Paris}}$ — and the outer product writes $v_{\text{Brasília}}$ in its place. Retrieval now returns the current fact, cleanly. The memory has become error-correcting: writes are replacements along the key’s direction, and $\beta_t$ lets the model choose, per token, between leaving the slot alone ($\beta\to 0$) and overwriting it ($\beta\to 1$).

One repair remains. The delta rule edits one key direction per step, so stale content on directions never revisited lingers forever. Gating adds learned, uniform forgetting — the matured form of the gate idea from lesson 3.3 — through a per-step decay $\alpha_t \in (0,1)$:

This is the Gated DeltaNet update of Yang, Kautz, and Hatamizadeh: precise targeted erasure from the delta rule, global fade from the gate, both computed from the token itself. The output side reads the edited board with a query, exactly as in linear attention.

Now the Qwen3.8-27B instantiation. Each DeltaNet layer runs the recurrence in heads, but with an asymmetry that inverts the one you know from attention: 16 QK heads and 48 V heads, all at head dimension 128. In the attention layers, grouped-query attention shares a few KV heads among many query heads; here, addressing is the shared commodity — each QK head’s keys and queries serve three value heads, so the layer maintains 48 separate $128\times 128$ state boards indexed by 16 addressing schemes. Content capacity is plentiful; addressing machinery is economized. Before any of this, the projected sequences pass through a short causal depthwise convolution with kernel 4, giving each update a cheap glimpse of the last few positions — local patterns like token bigrams need not consume the recurrent state at all.

Why give this mixer 48 of 64 layers? Because its cost profile is the mirror image of attention’s. The state per layer is a fixed set of $128\times128$ matrices — about 3 MiB per layer in the reference float32 path, a figure that does not grow whether the context holds one hundred tokens or 262,144 (lesson 7.2 puts exact numbers on this and on attention’s contrasting bill). Decoding advances the state in constant time per token. And because the update is linear in $S$, training can process sequences in parallel chunks rather than strictly step by step. What the fixed-size board cannot do is guarantee exact recall of an arbitrary token from half a million steps ago — it is a lossy, learned compression. Qwen therefore does not use DeltaNet everywhere: every fourth layer keeps full attention, and the division of labor between the two is the subject of lesson 4.16.

From the residual stream’s perspective, none of this drama is visible. The DeltaNet layer reads a normalized 5120-wide state, runs conv, projections, recurrence, and output gating, and adds back a 5120-wide update — the same contract every mixer honors. The checkpoint tensors you could not yet name in lesson 4.14 — the linear_attn projections, the kernel-4 convolution, the decay parameters — are precisely the machinery of $\alpha_t$, $\beta_t$, $k_t$, $v_t$, $q_t$ you can now read. What remains is the architectural question: given one mixer that is exact but expensive and one that is cheap but lossy, in what ratio do you mix them, and why three to one? That is the next lesson.