When training goes wrong: spikes, divergence, NaNs

1

Concept

Long pretraining runs eventually encounter unusual batches, transient hardware faults, collective timeouts, numerical overflow, and optimization regimes that pilots did not expose. Reliability begins by treating anomalies as evidence. Automatically skipping anything inconvenient may keep a graph moving while silently changing the data distribution or hiding a systemic failure.

A loss spike is a sudden rise followed by recovery or escalation. It may come from a genuinely difficult batch, corrupted tokenization, an abrupt mixture change, a learning-rate problem, exploding activations, a precision overflow, or a rank that supplied invalid values. The shape alone does not identify the cause. Compare token-level losses, per-rank values, gradient norms, activation norms, and the input manifest.

Divergence means the optimization trajectory stops returning to its previous range and worsens persistently. Excessive learning rate, inadequate warmup, bad initialization, optimizer-state corruption, repeated pathological data, or a precision recipe outside its stable range can contribute. A model can remain finite while already diverging, so waiting for NaN is too late.

NaN and infinity are terminal arithmetic signals, not diagnoses. Trace the first non-finite tensor with hooks or anomaly instrumentation. Softmax overflow, division by zero, invalid square roots, fp16 range, gradient reduction, and optimizer updates are candidates. Later layers spread the value rapidly, so the first reported NaN may be far downstream from the cause.

At every checkpoint, retain enough state for exact or close reproduction: model, optimizer, scheduler, loss scaler, random-number generators, data-loader cursor, mixture state, topology, code revision, and configuration. Log stable document and batch IDs without copying sensitive text into general logs. Preserve recent infrastructure events such as node replacement or network retry.

A forensic sequence starts from the last known-good checkpoint and replays the same batch. If failure reproduces, run it on one device and a wider dtype. Validate token ranges, masks, sequence boundaries, targets, and sample weights. If it does not reproduce, inspect nondeterminism, distributed collectives, memory corruption, and hardware telemetry. Compare healthy and failing ranks.

Recovery depends on cause. Repair or quarantine a provably corrupt sample with an auditable data-policy decision. Reduce loss scale for fp16 overflow. Restore optimizer state if it is corrupt. A temporary lower learning rate may recover some optimization spikes, but changing it permanently creates a new experiment. Rolling back weights without rolling back data position can either repeat or hide the trigger.

Preventive controls include gradient clipping, stable normalization, conservative initialization, warmup, finite-value checks at selected boundaries, input validation, checksum-protected checkpoints, and canary pilot runs. Too many synchronous checks can reduce throughput, so high-frequency cheap signals and lower-frequency deep probes should be designed together.

Loss curves should include annotations for checkpoint, restart, topology change, mixture transition, and learning-rate change. A gallery of real curves is useful only when provenance and axes are preserved; unlabeled screenshots invite storytelling. Compare tokens processed, not just wall-clock timestamps.

The durable operating principle is fail visibly and recover reproducibly. Spikes are symptoms, NaNs are propagation, and divergence is a trajectory. A serious training system makes each traceable to data, numerics, optimization, or infrastructure, then proves that the resumed run returns to an expected path.