Initialization, normalization, and residuals

1

Concept

Deep networks repeat transformations, so small scale problems compound. If every layer amplifies variance, activations and gradients can explode. If every layer shrinks it, signals vanish. Initialization, normalization, and residual connections attack different parts of this problem: the starting parameter scale, the evolving feature scale, and the length of the transformation path.

Initialization must break symmetry. If all units in a layer begin with identical weights, they receive identical gradients and remain duplicates. Random values break symmetry, but arbitrary variance is unsafe. Variance-aware schemes scale weights using fan-in and sometimes fan-out, the counts of incoming and outgoing connections. Xavier/Glorot initialization suits certain symmetric activations; He-style scaling accounts for ReLU discarding much of the negative side.

These formulas are starting assumptions, not guarantees. Activation, gating, residual depth, tied weights, and precision affect signal propagation. Biases are often initialized to zero because random weights already break symmetry, though specific gates may use deliberate bias values. Monitoring actual activation and gradient statistics is more reliable than treating an initializer name as proof.

LayerNorm operates across the feature dimension of one example or token. Given hidden vector $x$, it subtracts its mean, divides by its standard deviation plus an epsilon for stability, then applies learned scale and bias. Unlike BatchNorm, it does not require batch-wide statistics and behaves naturally for variable sequence lengths and autoregressive inference.

RMSNorm simplifies the operation by scaling according to root mean square without subtracting the feature mean. It retains learned scaling, and implementations vary in dtype and epsilon details. Saying it is “the same but faster” is too broad: speed depends on kernels and hardware, while removing mean-centering changes the mathematical operation. Architecture-level evidence should decide the trade-off.

A residual connection computes $y=x+F(x)$ when shapes match. If $F$ is initially small or later unnecessary, information can follow the identity path. Backward gradients also have a direct additive route instead of passing only through every transformation. The block learns a residual correction rather than recreating the entire representation from scratch.

Normalization placement creates pre-norm and post-norm designs. A simplified pre-norm block uses $x+F(\operatorname{Norm}(x))$; post-norm applies $\operatorname{Norm}(x+F(x))$. The choice changes the identity gradient path and the scale seen by sublayers. Pre-norm is often easier to optimize at depth, but final quality, residual scaling, and architecture details complicate universal claims.

These mechanisms work as a system. Good initialization makes the first update meaningful. Normalization constrains evolving feature scale. Residuals preserve routes through depth. None fixes a wrong objective, data leakage, or poor evaluation. Their value is infrastructural: they make deep composition trainable enough that attention and MLP blocks can learn useful transformations rather than spending the entire optimization budget fighting numerical collapse.

An empirical signal-flow check is straightforward. Before training, pass representative inputs through the model and record per-layer activation means, root-mean-square values, and gradient norms after one loss. Look for monotonic collapse, explosive growth, dead branches, and large discontinuities at residual additions. Repeat in the actual precision used for training. This does not replace theory, but it tests whether the initializer, norm epsilon, residual scaling, and implementation jointly realize the intended regime.