Loss functions and cross-entropy

1

Concept

A loss function converts a model’s output and a target into a scalar that optimization can reduce. The choice defines what counts as error during training. Mean squared error penalizes squared numerical distance and is natural for some regression settings. Cross-entropy evaluates predicted distributions and is central to classification and language modelling. Neither is a universal measure of product quality.

For multiclass classification, a network emits one logit $z_i$ per class. Logits are unrestricted scores. Softmax turns them into probabilities:

Adding the same constant to every logit changes no probability; only relative scores matter. The probabilities are positive and sum to one, matching a categorical outcome where exactly one class occurs.

If the observed class is $y$, one-hot cross-entropy simplifies to

Assigning the correct class probability near one yields loss near zero. Assigning it tiny probability yields a large loss. This asymmetry is useful: a confidently wrong prediction is penalized much more than an uncertain one. Cross-entropy is also differentiable with respect to logits, providing a dense learning signal for all classes.

For soft targets, $L=-\sum_i q_i\log p_i$, where $q$ is a target distribution. Label smoothing replaces an exact one-hot target with a distribution that reserves some mass for other classes. Knowledge distillation can use a teacher distribution. These choices change the learning problem; they should not be described as mere numerical tricks.

Numerical stability matters. Computing exponentials of large logits can overflow, while tiny softmax probabilities can underflow to zero before a logarithm. Libraries implement cross-entropy from logits using log-softmax and the log-sum-exp identity, usually subtracting the maximum logit. Calling softmax manually and then taking log is less stable and can accidentally apply softmax twice if the loss already expects logits.

Batch loss is usually a mean or sum over examples and, for language models, token positions. Padding and ignored labels need a mask so they contribute neither loss nor denominator incorrectly. Averaging per token gives each token equal weight; averaging per sequence gives sequences equal weight. The reduction is part of the objective and must be reported when lengths differ.

Class imbalance and asymmetric harm may require weighting, resampling, focal losses, or a separate decision policy. A lower average loss can coexist with worse performance on a rare critical class. Training loss also differs from evaluation metrics such as accuracy, F1, calibration error, or task success. Optimizing a differentiable surrogate is useful only when its relationship to the real decision is checked.

In a causal language model, the target at every position is the next token. One forward pass during training produces logits for many positions, and masked cross-entropy averages the negative log probability of observed continuations. The model learns a distribution, not a truth flag. Loss tells us how well probabilities fit the token dataset; broader claims about factuality, safety, usefulness, or reasoning require distinct evidence.

A practical loss audit uses hand-computable logits. Verify that raising the correct-class logit lowers loss, shifting all logits by one constant changes nothing, ignored positions contribute nothing, and the reported mean uses the intended denominator. These small invariants catch double-softmax, wrong label shifts, and padding bugs that a long decreasing curve can hide.