Residuals, pre-norm vs post-norm

01 · Concept

Concept

Here is a problem the next lesson will make vivid: Qwen3.8-27B stacks 64 layers, and they are not even all the same kind of layer — most mix tokens one way, a minority another way, and every one of them ends in the wide gated FFN you just met. How do sixty-four heterogeneous processing stages cooperate without any stage destroying what earlier stages built, and without gradients dying on the way back down? The answer is the architecture’s quietest and most load-bearing decision: a single continuous residual stream, 5120 numbers wide per token, that no layer ever replaces — layers only add to it.

For a transformation $F$, the core pattern is

The identity term $x$ gives information a direct route through the stack. A layer can learn a small useful correction instead of rebuilding the representation from scratch. During backpropagation, gradients also get an additive route that does not multiply through every sublayer’s Jacobian. This does not abolish exploding or vanishing gradients, but it makes optimizing genuinely deep stacks tractable — and 64 layers is genuinely deep.

Each layer writes twice. After its mixing sublayer, the stream holds its previous features plus a communication update; after the FFN, it holds that result plus a nonlinear position-wise update. The residual stream is therefore not any single component’s output. It is a shared workspace in which many overlapping features are superposed, and the fact that both a Gated DeltaNet layer and a full-attention layer can write to the same workspace — using the same interface, an added 5120-wide update — is exactly what lets Qwen mix layer types freely.

Normalization controls what each sublayer reads from that workspace. Classic LayerNorm computes a mean and variance across a token’s feature dimension, subtracts, divides, and applies a learned scale and shift. Qwen3.8-27B, like most modern decoders, uses RMSNorm: divide the vector by $\sqrt{\operatorname{mean}(x^2)+\epsilon}$ and apply a learned scale — no mean subtraction, no shift. In Qwen the epsilon is $10^{-6}$: small enough to be invisible for healthy activations, present inside the square root so a near-zero vector never divides by zero.

Where the normalization sits is the classic fork. The original Transformer used post-norm:

renormalizing the combined state after every addition. Walk the naive reasoning first, because it feels right: “normalize the final sum — surely the cleanest signal comes from calibrating the result.” But trace the gradient. In post-norm, the identity path itself passes through the normalization, so every backward step multiplies through a norm’s Jacobian; with 64 layers, that is 128 sublayer normalizations standing between the loss and the embedding. Deep post-norm stacks historically demanded careful warmup and were prone to divergence. The correction is pre-norm:

which is what Qwen3.8-27B does. Each sublayer receives a calibrated copy, but the main path remains a bare chain of additions — the light table never gets repainted. Concretely, one Qwen layer computes $x_1 = x + \operatorname{Mixer}(\operatorname{RMSNorm}(x))$ and then $x_2 = x_1 + \operatorname{FFN}(\operatorname{RMSNorm}(x_1))$, and the model repeats that contract 64 times before a final RMSNorm prepares the state for the output head.

Pre-norm has its own dynamics. Because the stream is never rescaled on the main path, its magnitude tends to grow as updates accumulate through depth; the per-sublayer RMSNorm hides that growth from each reader, and the final norm hides it from the head. Initialization, learning-rate schedule, precision, and optimizer all interact with these choices — pre-norm is an empirically friendlier default for very deep stacks, not a theorem about accuracy.

The phrase “residual stream” also disciplines interpretation. Because features are superposed in one vector space, a mixing layer’s output may be cancelled by a later update, carried unchanged for thirty layers, or transformed downstream. Looking at one sublayer’s output in isolation misses both the state already present in $x$ and the later operations that consume the sum.

A practical debugging habit: record three norms per sublayer — the stream $\lVert x\rVert$, the update $\lVert F(\cdot)\rVert$, and the sum. An update that dwarfs the stream can destabilize training; one that is always negligible may be a dead branch. These are diagnostics, not thresholds; expected scale depends on depth and architecture.

The enduring picture is collaborative editing at industrial depth. Qwen3.8-27B’s stream preserves the current document across 64 specialists of two different trades. RMSNorm hands each specialist a calibrated copy at a cost of one scale vector per sublayer, epsilon $10^{-6}$ guarding the division. Pre-norm placement keeps the master document continuous, which is precisely why the stack can be this deep — and this heterogeneous — without collapsing.