Initialization, normalization, and residuals

01 · Concept

Concept

Here is a question with an alarming answer. Qwen3.8-27B stacks 64 layers, each carrying a 5120-dimensional hidden state per token (Qwen3.8-27B Model Card, 2026). Suppose each layer, on average, multiplies the scale of what passes through it by 1.1 — a ten percent amplification, which sounds harmless. After 64 layers the signal has been multiplied by $1.1^{64}\approx446$. Now suppose the factor is 0.9 instead: $0.9^{64}\approx0.0012$, and the signal is gone. Depth compounds. A network is not forgiving of small scale errors; it raises them to the power of its own depth, in the forward pass and again in the backward pass, where the same products govern whether a gradient reaches layer one at all.

Three mechanisms manage this, each attacking a different part of the problem: initialization sets the starting scale, normalization controls the evolving scale, and residual connections shorten the path.

Initialization first, because it must do two jobs at once. It has to break symmetry — if every unit in a layer starts identical, they receive identical gradients and remain duplicates forever — and it has to control variance. Variance-aware schemes scale random weights by fan-in, the number of incoming connections, and sometimes fan-out: Xavier/Glorot for symmetric activations, He-style scaling to compensate for ReLU discarding roughly half the signal. These are starting assumptions, not guarantees, because gating, residual depth, and precision all perturb propagation. Biases are usually zero, since randomness in the weights already breaks symmetry.

Now the structural idea that made real depth possible. A residual connection computes

when shapes match. Read it as a claim about responsibility: the block’s job is to compute a correction to be added to what already exists, not to reconstruct the representation from scratch. If $F$ contributes nothing useful, information still flows through the identity path untouched. Backward, the derivative of $y$ with respect to $x$ contains an additive identity term, so gradient reaches earlier layers without having to survive the product of every intervening Jacobian.

This creates what practitioners call the residual stream: one 5120-wide vector per token that enters at the embedding, travels the full depth of the model, and is written to by every sublayer along the way. Each attention or feed-forward block reads from it, computes something, and adds the result back. Nothing overwrites it. The stream is the model’s working memory, and the 64 layers are 128 successive contributions to it.

Which raises the scale question again, in a sharper form. If a fresh block adds a contribution of comparable magnitude to the stream at every step and the contributions are roughly independent, the stream’s variance accumulates additively — meaning its root-mean-square magnitude grows like $\sqrt{\text{depth}}$, so by layer 64 it is around eight times what it was at layer one. Sublayers deep in the stack would receive inputs an order of magnitude larger than sublayers near the input, and every weight matrix would have to cope with wildly different operating scales depending on where it happened to sit.

Normalization is the answer, and its placement is the whole design. A pre-norm block computes

which is what Qwen3.8-27B uses (Qwen3.8-27B Model Card, 2026). The normalization sits on the branch, not on the stream. Each sublayer receives a rescaled copy of the hidden state — always at a familiar magnitude, whether it sits at layer 3 or layer 61 — while the stream itself continues down the identity path with no operation applied to it at all.

The specific normalizer is RMSNorm. For a hidden vector $x$ with $d$ entries it computes

with a learned per-feature gain $g$ and a small stabiliser $\varepsilon$, conventionally around $10^{-6}$. Walk one vector through. Say a token’s hidden state at layer 40 has a root-mean-square value of 12.0 across its 5120 entries. The denominator is $\sqrt{144+10^{-6}}=12.0$ to any precision that matters, so every entry is divided by 12 and the sublayer sees a vector of unit RMS. Now consider a near-dead position whose entries are all around $10^{-5}$: the mean square is $10^{-10}$, which is ten thousand times smaller than $\varepsilon$, so the epsilon dominates the square root and the output is scaled down toward zero rather than exploding through a division by almost nothing. That is precisely the job epsilon does — it is not a rounding fudge, it is the guard that keeps the operation defined at the bottom of the range.

Note what RMSNorm does not do: it never subtracts the mean. LayerNorm centers the feature vector before scaling; RMSNorm only rescales.

The classic wrong turn here is placement. The original Transformer used post-norm, $y=\operatorname{Norm}(x+F(x))$, which puts a normalization operation on the stream itself. It works, and at moderate depth it works well. But it destroys the clean identity path: gradient flowing backward must now pass through a normalization at every one of the 64 steps, and the deep stacks that result are notoriously fragile to train, needing careful warmup and sometimes failing outright. Moving the norm onto the branch — the one-line change from post-norm to pre-norm — restores the unobstructed additive highway, and it is why pre-norm is the default in essentially every large model shipped since.

These mechanisms are infrastructure, and it matters that you read them as such. None of them decides what the model learns. They make deep composition numerically survivable so that the attention and feed-forward blocks of track 4 can spend the optimization budget learning useful transformations rather than fighting collapse. Lesson 4.10 returns to this arrangement with the full transformer block in view.

An empirical check is cheap and worth building early. Before training, push representative inputs through the model and record per-layer activation means, root-mean-square values, and gradient norms after one loss. Look for monotonic collapse, explosive growth, dead branches, and discontinuities at residual additions — and run it in the precision you will actually train in, since bf16 rounding is part of the system you are testing.