LSTM and GRU: gates that remember

1

Concept

LSTM and GRU cells modify recurrent computation so information can follow an additive path controlled by learned gates. A gate is usually a sigmoid output between zero and one, multiplied elementwise with a vector. It can softly preserve, suppress, or blend features based on the current input and previous state. The gate values are learned consequences of the prediction objective, not hand-assigned semantic switches.

An LSTM maintains a cell state $c_t$ in addition to hidden output $h_t$. A common formulation computes forget gate $f_t$, input gate $i_t$, output gate $o_t$, and candidate $\tilde c_t$. Its central update is

followed by $h_t=o_t\odot\tanh(c_t)$. All gates depend on $x_t$ and $h_{t-1}$ through learned affine maps.

The forget term decides how much of each previous cell coordinate remains. The input term controls how much candidate content is written. The output gate controls what transformed cell content appears as the hidden state for predictions and the next recurrence. Because the cell update is additive, a component with forget gate near one can carry value and gradient across steps without repeatedly passing through a full nonlinear state transformation.

This is a better route, not perfect memory. Gates can learn values below one, causing exponential decay. The finite-dimensional cell can face competing information. Saturated gates may learn slowly, and long sequences still require correct credit assignment. Bias initialization sometimes encourages early remembering, but the appropriate policy depends on implementation and task.

A GRU merges cell and hidden state. Its update gate controls interpolation between the previous state and a candidate; its reset gate controls how much previous state contributes while constructing that candidate. Exact equations vary by convention, including whether reset is applied before or after a matrix multiplication. These variants are not always numerically equivalent, so checkpoint compatibility requires the precise definition.

GRUs have fewer gates and parameters than comparable LSTMs, which can reduce computation. LSTMs expose a separate cell pathway and may offer more control. Neither universally wins. Dataset size, sequence structure, hidden width, implementation quality, and tuning influence results. A fair comparison matches parameter or compute budget and reports recurrent-cell equations.

Gate visualizations can be informative but easy to anthropomorphize. A high forget value on a token does not prove the unit stores a human-named fact. State features are distributed, gates interact across coordinates, and downstream transformations can overwrite or ignore preserved content. Causal interventions and task-level evaluation provide stronger evidence than a colorful heatmap alone.

Gated recurrence marked a major improvement for speech, translation, time series, and language modelling. Transformers later traded recurrent compression for attention-based access and greater training parallelism, but gates did not disappear: gated MLPs such as SwiGLU use related multiplicative control. The durable insight is that additive state plus learned valves can create trainable information highways through a long computation.

To compare cells honestly, match the total parameter count by adjusting hidden width, then report latency and memory at the sequence lengths of interest. Equal hidden size favors the smaller GRU in parameter cost, while equal parameters changes width and representation capacity. Also test streaming state resets explicitly: carrying state across unrelated sequences can create impressive-looking leakage rather than memory.