Linear attention and fast weights

01 · Concept

Concept

Attention pays for its memory in the most literal way possible: it keeps everything. To produce token $t$, a causal attention layer compares the new query against the key of every earlier position and mixes their values:

Count what this step actually costs. The naive plan — the one every attention decoder follows — stores all $t$ past key and value vectors and performs $t$ similarity computations for the new token. Token one costs one comparison, token one thousand costs a thousand, token $n$ costs $n$: per-token work is $O(n)$ in context length, and the stored keys and values grow without bound. That store is the KV cache, and lesson 7.2 derives its price for our course model: in Qwen3.8-27B at its native 262,144-token context, the cached keys and values of the attention layers reach 16 GiB for a single sequence. The layer retains every past key-value record and gives the new query a direct weighted path to each one. That avoids fixed-state compression, but softmax still forms a weighted mixture and does not guarantee exact retrieval of an arbitrary stored token.

Lesson 3.2 showed the opposite design: an RNN carries one fixed-size hidden state, pays the same cost at every step, and stores nothing per token. The question this lesson answers is whether attention’s addressing behavior can live inside a fixed-size state.

Katharopoulos et al. (2020) noticed that the obstacle is the softmax. Replace the exponential similarity with a kernel feature map $\phi$, so the weight on position $i$ is proportional to $\phi(q_t)^{\top}\phi(k_i)$, and the sum becomes associative:

Both sums no longer depend on the query, so they can be maintained incrementally. Define a matrix-valued state and a normalizer:

and the output is one read, $o_t=\dfrac{S_t^{\top}\phi(q_t)}{z_t^{\top}\phi(q_t)}$. Look at what happened: this is a recurrence. Same update rule at every position, fixed-size state, constant work per token — the RNN blueprint from lesson 3.2, except the hidden state is now a $d\times d$ matrix of key-to-value associations instead of a vector. Per-token cost drops from $O(n)$ to $O(1)$, and memory from unbounded to constant.

Here is the classic wrong turn. The additive update only ever adds. Write value $v_a$ under some key, later write a different $v_b$ under a nearly identical key, and retrieval returns a blend of both; the stale association is never removed, and as thousands of writes superpose, interference grows. Early linear attention traded away softmax attention’s precision and underperformed it on tasks needing exact recall — which is why it spent years as the “losing” idea. The tempting conclusion is that this is fundamental: a fixed-size state cannot forget selectively, so lossy blur is the unavoidable price of $O(1)$. That conclusion is wrong, and the fix was sitting in decades-old associative-memory theory: change the write, not the size.

The delta rule makes the write error-correcting. Before writing, retrieve what the state currently predicts for the incoming key — read it the same way any query is read, $S_{t-1}^{\top}\phi(k_t)$ — and write only the difference:

with $\beta_t\in(0,1)$ a learned write strength. Walk through the two cases. If the state already stores $v_t$ for this key, the error is near zero and the state is left alone. If it stores something stale, the update uses that error to change the state along the incoming key direction — an error-correcting write, not an exact overwrite, whose effect depends on $\beta_t$, the key features, and interference from other associations. Each step is one step of online gradient descent on the memory’s own retrieval error: the state learns, while the model runs. DeltaNet is linear attention with this write; adding the learned forgetting gates of lesson 3.3 — a decay on $S_{t-1}$ so the state can also fade what no longer matters — yields Gated DeltaNet (Yang et al., 2024), which lesson 4.15 dissects in full.

And that difference is precisely how the losing idea of 2020 became production architecture in 2026. Qwen3.8-27B, this course’s specimen, builds 48 of its 64 layers as Gated DeltaNet and keeps full softmax attention in only 16, one in every four (Qwen3.8-27B Model Card, 2026). On the current Transformers reference float32 path, each DeltaNet layer.s recurrent state is about 3 MiB; across all 48 layers that is roughly 144 MiB — constant no matter whether the sequence holds one hundred tokens or 262,144 — against the 16 GiB the attention layers’ KV cache reaches at full context. The hybrid is an engineering confession that both function classes earn their place: fixed-size error-correcting memory for cheap length, a few per-position key-value access layers for lookups that fixed-state compression cannot represent reliably. The RNN did not lose to attention after all; it went away, learned to write like a memory, and came back as most of the model.