Loss functions and cross-entropy

01 · Concept

Concept

A model has just read the prefix “The capital of France is” and must produce a number saying how well it did. What number? “Right or wrong” is nearly useless: it cannot distinguish a model that ranked the correct continuation second from one that ranked it two hundred thousandth, and a flag has no gradient to descend. The scoring rule has to reward probability mass placed on what actually happened, and it has to punish confident mistakes harder than uncertain ones. That rule is cross-entropy, and this lesson makes it concrete at the scale the rest of the course works in.

A loss function turns model output and target into a single scalar that optimization can reduce. For classification the network emits one logit $z_i$ per class — an unrestricted score, not a probability. Softmax converts the logits into a distribution:

Adding the same constant to every logit changes nothing, so only relative scores matter. If the observed class is $y$, one-hot cross-entropy is simply

Probability near one gives loss near zero; probability near zero gives an enormous loss. The asymmetry is the useful part.

Now scale it. In Qwen3.8-27B the classes are vocabulary entries, and there are 248,320 of them (Qwen3.8-27B Model Card, 2026). Every token position is a 248,320-way classification, and the final linear layer that produces those logits is a matrix mapping the 5120-dimensional hidden state to 248,320 outputs — about 1.271 billion parameters, held separately from the embedding table because this model does not tie the two.

Work out what the numbers mean, step by step. First establish the baseline: a model that has learned nothing spreads probability uniformly, giving every entry $1/248{,}320$, so its loss is $-\log(1/248{,}320)=\ln(248{,}320)\approx12.42$ nats per token. That is the uniform baseline, the cost of pure ignorance; it is not a ceiling, because assigning the observed token still less probability produces arbitrarily larger loss. Now suppose training brings the average loss to 2.0 nats. Exponentiate to get perplexity: $e^{2.0}\approx7.4$. Read that as an effective branching factor — on average the model behaves as though it were choosing uniformly among about seven or eight candidates rather than a quarter of a million. The 12.42 nats of raw uncertainty have been cut to 2.0.

Keep going, because the geometry of nats is not linear. Reaching 1.9 nats sounds like a rounding difference; in perplexity it is $e^{1.9}\approx6.7$, a reduction of nearly ten percent in effective candidates. Each nat removed divides the branching factor by $e\approx2.72$. This is why loss curves that look flat late in training still represent real progress, and equally why a tenth of a nat is worth arguing about.

Here is the classic wrong turn, and it survives into published comparisons. Two models report average cross-entropy of 2.0 and 2.3 nats. It is tempting to conclude the first is better. It may not be comparable at all: if the second model uses a coarser tokenizer, its tokens carry more text each, so it makes fewer, harder predictions over the same document. Loss per token is not loss per unit of text. The correction is to normalize by something tokenizer-independent — bits per byte or bits per character, as lesson 1.7 set up — before comparing across vocabularies. Within one tokenizer, loss per token is exactly the right currency; across tokenizers, it is a category error.

Numerical stability is the other place implementations go wrong. Exponentiating large logits overflows, and tiny softmax probabilities underflow to zero before a logarithm can be taken. Libraries therefore fuse log-softmax with negative log-likelihood using the log-sum-exp identity, usually by subtracting the maximum logit first. Two failure modes follow from ignoring this: passing already-softmaxed probabilities into a primitive that expects logits, which silently applies softmax twice and flattens the distribution, and taking the log of a materialized zero, which yields an infinity that propagates through the whole batch.

Two details of reduction deserve care because they quietly change the objective. Batch loss is a mean or sum over examples and, for language models, over token positions; padded positions and ignored labels need a mask so they contribute neither to the numerator nor the denominator. Averaging per token weights every token equally, while averaging per sequence weights every sequence equally — different objectives, and the difference is visible whenever lengths vary. Soft targets change the problem again: with a target distribution $q$ the loss becomes $L=-\sum_i q_i\log p_i$, which is how label smoothing and knowledge distillation work. These are modelling decisions, not numerical conveniences.

A practical audit uses logits you can compute by hand. Verify that raising the correct-class logit lowers the loss, that shifting all logits by one constant changes nothing, that ignored positions contribute nothing, and that the reported mean uses the denominator you intended. Those four invariants catch double-softmax, off-by-one label shifts, and padding bugs that a long, smoothly decreasing curve will otherwise hide for days.