Residuals, pre-norm vs post-norm

1

Concept

A Transformer does not pass each layer’s output into the next as a total replacement. It maintains a residual stream and asks each sublayer to add an update. For a transformation $F$, the core pattern is

The identity term $x$ gives information a direct route through the stack. A layer can learn a small useful correction instead of rebuilding the representation. During backpropagation, gradients also have an additive route that does not multiply through every sublayer’s Jacobian. This does not eliminate exploding or vanishing gradients, but it makes deep optimization substantially more manageable.

Attention and the FFN each write into this stream. After attention, the state contains its previous features plus a communication update. After the FFN, it contains that result plus a nonlinear position-wise update. The residual stream is therefore not one component’s output. It is a shared workspace accumulating many overlapping features.

Normalization controls the scale and distribution seen by transformations. LayerNorm computes statistics across a token’s feature dimension, then applies learned scale and shift. Many modern decoder models use RMSNorm, which normalizes root-mean-square magnitude without subtracting the mean. The placement question remains similar: normalize before a sublayer or after its residual addition?

The original Transformer used post-norm:

The combined state is normalized after every update. A common modern alternative is pre-norm:

Here each sublayer receives normalized input, but the main residual path remains an identity addition. Deep pre-norm stacks often optimize more reliably because the gradient has a cleaner route through successive additions. This is an empirical architectural tendency, not a proof that pre-norm is always more accurate.

Post-norm and pre-norm also produce different representational dynamics. Post-norm repeatedly rescales the combined stream. Pre-norm allows residual magnitude to grow through accumulated updates, so architectures may use a final normalization and sometimes explicit residual scaling. Initialization, depth, learning-rate schedule, precision, and optimizer all interact with the choice.

Dropout, when used, normally applies to the sublayer branch before addition, not to the identity path. That preserves a reliable route while regularizing updates. At inference dropout is disabled. Other methods such as stochastic depth can remove entire residual branches during training, again relying on the identity path to keep the network valid.

The phrase “residual stream” helps interpretation. Many features are superposed in the same vector space, and components read from and write to that space. An attention head’s output may be cancelled by another update, carried unchanged across layers, or transformed later. Looking only at one sublayer output misses the state already present in $x$ and the later operations that consume the sum.

A simple debugging habit is to record norms: the norm of $x$, the norm of $F(\cdot)$, and the norm after addition. An update that dwarfs the stream can destabilize training; an update that is always negligible may indicate a dead branch. These are diagnostics, not universal thresholds, because expected scale depends on architecture.

The enduring picture is collaborative editing. The residual stream preserves the current document. Each normalized sublayer reads it and proposes a structured change. Normalization placement determines whether calibration happens before the specialist works or after the edit is merged, changing how safely very deep stacks can be trained.