RNNs, fixed-size state, and the vanishing gradient

01 · Concept

Concept

Lesson 3.1 posed the state question and sketched two extreme answers. This lesson takes the fixed-size answer seriously, finds out exactly why it lost the 2017 argument, and then reports something surprising: the design that lost now runs in three quarters of the layers of this course’s model.

A recurrent neural network is the minimal implementation of a bounded carried state:

One transition, reused at every position. Read it as a sequence mixer: information from position 12 reaches position 40,000 only by having survived inside $h$ through 39,988 applications of the same function. Memory is constant, work per token is constant, and nothing is stored per position. Lesson 3.1’s arithmetic already told you the price — the bounded channel — so this lesson is about the second price, which turned out to be worse.

Unroll the equations and something important becomes visible: an RNN over a sequence of length $T$ is a network of depth $T$ whose layers share parameters. Sequence time and network depth are the same axis. Everything lesson 2.8 said about compounding scale across 64 layers applies here across 40,000 steps.

To learn that an early token caused a late error, training uses backpropagation through time: unroll the graph, differentiate, and accumulate gradient contributions from every use of the shared matrices. The derivative from state $h_T$ back to state $h_k$ contains one Jacobian per intervening step, multiplied together.

Put numbers on it, in the simplest possible setting. Take a scalar recurrence $h_t=\tanh(w\,h_{t-1}+x_t)$. Each step contributes a factor of $w\cdot\tanh'(\cdot)$, and since $\tanh'$ is at most one, the factor is at most $|w|$. Set $w=0.5$ and look across 50 steps: the product is at most $0.5^{50}\approx8.9\times10^{-16}$. That value remains representable in bf16, whose exponent range matches float32, but it is an extremely small relative learning signal and can be overwhelmed when accumulated with much larger contributions. The early state receives almost no usable information about how it should change to reduce a loss fifty steps later — the vanishing-gradient problem. Now set $w=1.5$ instead. The product reaches $1.5^{50}\approx6.4\times10^{8}$, so an update can become large enough to destabilize the run: exploding gradients. Between these lies a narrow band, and $\tanh$ makes it narrower still, because a saturated unit contributes a derivative near zero and drags the product down even when $w$ is well chosen.

Note carefully what has and has not failed. The forward state may still carry usable information about the distant past. What breaks is credit assignment: training cannot discover which early computation to change. This is why the problem is invisible in a forward-pass diagram and fatal in practice.

Here is the classic wrong turn, and it is nearly universal among people meeting these two failures for the first time. Training explodes, you add gradient clipping, the run stabilizes, and you conclude the gradient pathology is handled. It is not. Gradient clipping rescales a gradient whose norm exceeds a threshold — it acts after differentiation, on the magnitude. With standard norm clipping, the scale factor is capped at one: large gradients shrink, while small gradients are left unchanged. That controls explosion, and it is standard practice for good reason, but it does nothing for vanishing because it cannot reconstruct information already attenuated by the long Jacobian product. The diagnostic that separates the two: run a synthetic task with a single relevant marker placed at increasing distances from the prediction, and plot both accuracy and the gradient norm at the marker against distance. If the curve collapses toward chance with clipping enabled, distant credit was never restored, and no amount of clipping threshold tuning will change that.

Mitigations that do address vanishing attack the product itself rather than its output. Orthogonal initialization keeps the recurrent Jacobian near norm-preserving at the start. Truncated backpropagation shortens the unrolled window, which saves memory and compute while explicitly capping how far credit can travel — a real limitation that belongs in the reported model, not a free optimization. Additive state paths, which lesson 3.3 builds with gates, let a component travel across steps without passing through a full nonlinear transformation each time. And attention, arriving in lesson 3.5, sidesteps the chain entirely by creating a direct edge between distant positions.

Attention won, and it won on two counts at once: it removed the long product from the gradient path, and it made training parallel across positions, since every position’s representation can be computed simultaneously rather than waiting for its predecessor. Sequential execution was arguably the more decisive defeat — a design that cannot saturate a modern accelerator during training loses regardless of its other merits.

The arithmetic that makes it worth repairing is the one from lesson 3.1. A fixed-size state costs the same at token one and at token 262,144; in Qwen3.8-27B the recurrent layers hold roughly 144 MiB of reference float32 state in total, flat in sequence length, while the attention layers’ stored keys and values reach 16 GiB at full context, as lesson 7.2 derives. That gap is why anyone bothered to fix the RNN, and it is worth carrying into the next four lessons.