When training goes wrong: spikes, divergence, NaNs

01 · Concept

Concept

Two weeks into a run, the dashboard shows the averaged training loss stepping from 2.05 to 2.59 and staying there. It is not dramatic. Nobody would call it a catastrophe. The on-call engineer notes it, sees no NaN, and lets the job continue. Six hours and a considerable amount of cluster time later, someone opens the per-rank view and finds that 63 ranks are training normally at 2.0 while rank 41 has been sitting at 40.0 since the moment of the step. The global mean was never lying; it was averaging.

Do that arithmetic, because it is the whole reason per-rank telemetry exists:

One rank in complete failure moves the headline number by half a nat. Sixteen times more ranks, and it would move it by three hundredths. Averaging is a low-pass filter on exactly the signal you most need, and the larger the job, the more effectively it hides the thing that killed it.

The second half of reading anomalies well is knowing what a nat means. From lesson 5.1, mean cross-entropy is the mean negative log-probability of the correct tokens, so $e^{-L}$ is their geometric-mean probability. Uniform prediction over 248,320 classes gives $\ln 248{,}320\approx 12.42$ nats: a useful reference baseline, not a ceiling. NLL has no finite upper bound because the correct-token probability can approach zero. A jump from 2.05 to 2.90 moves the geometric mean from $e^{-2.05}\approx 0.129$ to $e^{-2.90}\approx 0.055$. That quantifies the change but does not diagnose it. Distance from the uniform baseline cannot by itself distinguish a reset, corrupted batch, rank failure, or genuine optimization divergence.

With that calibration in place, the vocabulary of failures becomes usable. A loss spike is a sudden rise followed by recovery or escalation. It can come from a genuinely difficult batch, corrupted tokenization, an abrupt mixture change, a scheduler problem, exploding activations, a precision overflow, or a rank feeding invalid values downstream. The shape alone identifies none of these; compare token-level losses, per-rank values, gradient norms, activation norms, and the input manifest before forming a hypothesis. Spikes aligned exactly with a restart point at scheduler, optimizer, or data-cursor state rather than at the data, which is why lesson 5.11 insisted that scheduler position is checkpoint state.

Divergence means the trajectory stops returning to its previous range and worsens persistently. Excessive learning rate, inadequate warmup, poor initialization, corrupted optimizer state, repeated pathological data, or a precision recipe pushed outside its stable range all contribute. Crucially, a model can remain entirely finite while already diverging — which makes “wait until something becomes NaN” a detector that fires long after the run was worth saving.

NaN and infinity are terminal arithmetic signals, not diagnoses. Trace the first non-finite tensor with hooks or anomaly instrumentation rather than reasoning backwards from where it surfaced, because later layers spread the value within a single forward pass and the first report is usually far downstream of the cause. Softmax overflow, division by zero, invalid square roots, fp16 range, gradient reduction, and optimizer updates are all candidates. Under the mixed-precision recipe of lesson 5.9 the fp16 case is mechanical: the format’s maximum is 65,504, so with a loss scale of 65,536 any gradient above roughly 1.0 overflows to infinity, and the scaler’s job is to detect that, skip the step, and halve the scale to 32,768 — which is a controlled response, not an incident.

Architecture is part of the stability story, and this model’s config shows the devices plainly. RMSNorm with epsilon 1e-6 in a pre-norm residual stream is the baseline: normalizing before each sublayer stabilizes the scale seen by that sublayer and improves gradient behavior. The residual stream itself is not normalized after each addition and may grow with depth, as lesson 4.10 explains; pre-norm does not bound it. The gated feed-forward multiplies a SiLU-activated gate against the up-projection before the 17,408-wide intermediate is projected back down, so a block can attenuate its own contribution rather than being forced to emit something. And the gate in Gated DeltaNet governs how much each step writes into and decays out of the fixed-size recurrent state — precisely the control that lesson 3.2 identified as missing from naive recurrent networks, whose states grew or vanished without bound. A hybrid stack that carries recurrent state through 48 of its 64 layers would be a stability nightmare without it.

The forensic sequence itself is unglamorous and works. At every checkpoint, retain enough to reproduce closely: model, optimizer, scheduler, loss scaler, random-number generators, data-loader cursor, mixture state, topology, code revision, configuration, and recent infrastructure events such as node replacement or network retry. Log stable document and batch identifiers without copying sensitive text into general logs. When something fires, start from the last known-good checkpoint and replay the same batch. If the failure reproduces, rerun it on one device in a wider dtype and validate token ranges, masks, sequence boundaries, targets, and sample weights. If it does not reproduce, look at nondeterminism, collectives, memory corruption, and hardware telemetry, and compare a healthy rank against the failing one.

Recovery depends on which of those it was. Quarantine a provably corrupt sample under an auditable data policy. Lower the loss scale for an fp16 overflow. Restore optimizer state if it is corrupt. A temporary reduction in learning rate can carry a run past an optimization spike, but making it permanent creates a new experiment that must be described as one. Rolling back weights without rolling back the data cursor either repeats the trigger or conceals it.

Preventive controls are cheaper than any of this: gradient clipping, stable normalization, conservative initialization, adequate warmup, finite-value checks at selected boundaries, input validation, checksum-protected checkpoints, and canary pilots. Too many synchronous checks cost throughput, so pair cheap high-frequency signals with occasional deep probes. Annotate loss curves with checkpoints, restarts, topology changes, mixture transitions, and rate changes, and compare against tokens processed rather than wall-clock timestamps.

The operating principle is to fail visibly and recover reproducibly. Spikes are symptoms, non-finite values are propagation, divergence is a trajectory — and a serious training system can trace each of them to data, numerics, optimization, or infrastructure, then demonstrate that the resumed run returns to the path it was on.