RNNs and the vanishing gradient

1

Concept

A simple recurrent neural network updates one hidden state as it reads a sequence:

The same matrices are reused at every position. Unrolling the equations reveals a deep network whose layers share parameters, with sequence time acting like depth.

The hidden state offers memory, but every new update transforms the old state. To learn that an early token caused a much later error, training uses backpropagation through time. The computational graph is unrolled, and gradient contributions from every use of the shared parameters are accumulated. Exact full unrolling costs memory proportional to sequence length.

The derivative from a later state to an earlier one contains a product of recurrent Jacobians. In a simplified linear recurrence, powers of $W_{hh}$ appear directly. If relevant directions repeatedly contract by factors below one, the product approaches zero: the vanishing-gradient problem. If they expand above one, the product can grow rapidly: exploding gradients. Nonlinear derivatives multiply into the same chain.

Sigmoid and tanh saturate when pre-activations have large magnitude. In saturated regions their derivatives are small, adding another shrinking factor at each step. A vanishing gradient means early states receive almost no information about how they should change to reduce a distant loss. The forward state may still carry some signal; the critical failure is that training cannot assign long-range credit effectively.

Exploding gradients cause huge, unstable updates and non-finite values. Gradient clipping rescales or caps a gradient when its norm exceeds a threshold. This protects optimization and is widely useful, but it treats the magnitude after differentiation. It does not create a learning signal where repeated derivatives already erased one.

Truncated backpropagation limits the unrolled window, saving memory and compute. State may continue forward across chunks while gradients stop at boundaries. This explicitly limits credit assignment and can bias learning, although it may be practical when relevant dependencies are mostly local. The chunking and state-detachment policy should be part of the reported model.

Initialization can keep recurrent dynamics near a stable scale, and orthogonal matrices help preserve norms in simplified settings. Normalization and residual or skip connections can improve paths. LSTM and GRU cells introduce additive, gated state routes designed to retain information and gradients. Attention later supplies direct edges between distant positions, shortening the path dramatically.

RNNs remain useful when streaming state, bounded memory, or low per-step compute matters. Their limitation is not that they categorically forget everything, but that repeated transformations make learning long dependencies fragile and sequential execution restricts training parallelism. The unrolled view unifies both facts: one elegant shared transition becomes a very deep chain, and the chain’s Jacobian geometry controls what the past can teach the future.

A diagnostic synthetic task can place one relevant marker at increasing distances from the prediction. Plot accuracy and the gradient norm at the marker against distance. This separates a vague claim of “long memory” from measurable degradation. Run the same protocol with and without clipping: clipping may stabilize exploding runs, but a curve still collapsing toward chance reveals that distant credit was not restored.