Entropy, perplexity, and what 'good' means

1

Concept

Information theory turns probability into a measure of surprise. An event assigned probability $p$ has self-information $-\log p$: likely events are unsurprising, while unlikely events carry more information when they occur. The logarithm’s base chooses units. Base two yields bits; the natural logarithm yields nats. Machine-learning libraries commonly use natural logs.

The entropy of a discrete distribution $p$ is expected surprise:

A distribution concentrated on one outcome has zero entropy. A uniform distribution over many outcomes has higher entropy. Entropy describes uncertainty in the distribution itself. It is not automatically disorder in the everyday sense, nor does high entropy imply bad modelling when the underlying situation is genuinely ambiguous.

To evaluate a model distribution $q$ on outcomes generated by $p$, use cross-entropy:

On a finite language dataset, the observed next token stands in for the data distribution. We average the negative log probability assigned to each correct token. This is mean negative log-likelihood, the familiar token-level training loss. Assigning tiny probability to the observed token incurs a large penalty; assigning high probability incurs a small one.

Cross-entropy decomposes as $H(p,q)=H(p)+D_{KL}(p\Vert q)$. The first term is uncertainty inherent in the data distribution; the KL divergence is the extra cost of using $q$ instead of $p$. We do not know the true language distribution exactly, and a test corpus is only a sample, but the decomposition explains why a perfect model cannot remove genuine uncertainty.

Perplexity exponentiates average cross-entropy. With natural logs, $\operatorname{PPL}=\exp(H)$. If the model faced an equally likely choice among $k$ tokens at every step, perplexity would be $k$. Real distributions are not uniform, so “effective branching factor” is an intuition, not a literal count of candidates. Lower perplexity means the model assigned more probability to observed tokens under that protocol.

Comparability has strict boundaries. Perplexity depends on the dataset, tokenization, treatment of beginning and ending tokens, context length, sliding-window method, and whether loss is masked on some positions. A tokenizer that splits text into different units changes the events being averaged. Two reported numbers from different vocabularies cannot be ranked naively. Byte- or character-normalized measures can help, but their definition must be explicit.

Data leakage can make perplexity deceptively low. Evaluation text duplicated in training tests memory as well as generalization. Domain mismatch can make a strong model look weak for the intended application or vice versa. Statistical uncertainty matters when differences are small. Report corpus provenance, token count, model and tokenizer versions, context policy, and enough precision to support—not exaggerate—the conclusion.

Finally, predictive fit is not the whole meaning of “good.” A model with lower perplexity may still follow instructions poorly, hallucinate, express harmful biases, or cost too much to serve. Post-training can improve assistant behavior while changing base-language metrics in complex ways. Perplexity is a clean answer to a narrow question: how surprised was this model by these tokens? It is valuable precisely when that question is kept separate from broader claims about usefulness and reliability.