Mixed precision: fp16, bf16, fp8

1

Concept

Training stores and moves enormous numbers of activations, weights, gradients, and optimizer values. Smaller numeric formats reduce memory traffic and can unlock faster accelerator units. The word mixed matters: robust training does not blindly cast every value to the narrowest dtype. It assigns precision according to numerical risk.

A floating-point value divides bits among sign, exponent, and fraction. More exponent bits increase dynamic range; more fraction bits improve local precision. fp16 uses five exponent bits and ten explicit fraction bits. It represents values more precisely than bf16 around the same magnitude, but its limited exponent range makes overflow and underflow common in training.

Dynamic loss scaling protects small fp16 gradients. Multiply the loss by a scale before backward, creating larger representable gradients. Before the optimizer step, unscale them and check for non-finite values. If overflow occurred, skip the update and reduce the scale; after stable steps, cautiously increase it. In exact arithmetic scaling would cancel out, but finite representation makes the path matter.

bf16 uses eight exponent bits, the same exponent width as fp32, and seven explicit fraction bits. Its range makes overflow and underflow less troublesome, while its coarser mantissa increases rounding. On supporting hardware, bf16 is often a forgiving training default. Accumulations and selected operations still commonly use fp32.

fp8 uses eight total bits and comes in formats that trade exponent range against fraction precision, commonly described as E4M3 and E5M2. It can greatly reduce storage and speed supported matrix operations, but raw casting is unsafe. Training recipes maintain scales so tensor values occupy the useful representable range, choose formats by operation, accumulate in wider precision, and monitor saturation.

Scaling can be per tensor, channel, or block. A single tensor-wide scale is cheap but lets one outlier waste most of the range. Finer scales represent varied distributions better but require more metadata and kernels. Delayed scaling may use recent activation maxima to choose a scale for later steps, introducing a tradeoff between responsiveness and stability.

Sensitive operations often remain wider: reductions, softmax normalization, loss calculation, norm statistics, optimizer moments, and master weights. Exact choices depend on framework and architecture. Numerically stable kernels may accept low-precision inputs but internally accumulate in fp32. Inspect operation policy rather than inferring it from the model parameter dtype.

Distributed training adds concerns. A gradient can be finite on each rank before reduction yet overflow during accumulation. Communication may use a compressed dtype distinct from optimizer storage. Scales must be synchronized where tensors are logically shared. Checkpoints need enough metadata to resume the precision recipe, including the gradient scaler.

Validation compares against a trusted wider-precision baseline. Monitor loss curves, gradient norms, skipped steps, overflow counts, saturation, and downstream metrics. One short run without NaNs is weak evidence; small rounding differences can change optimization gradually. Determinism across dtypes is not expected, so compare distributions and outcomes, not bit identity.

The stable rule is to spend bits where they protect information. Narrow formats accelerate large, tolerant matrix work. Wider formats accumulate many contributions and preserve sensitive state. Mixed precision succeeds when the boundaries are explicit, measured, and checkpointed—not when a global cast happens to survive.