Entropy, perplexity, and what 'good' means

01 · Concept

Concept

Two teams evaluate language models on the same corpus. One reports a perplexity of 7.4, the other 4.2, and the second declares victory. Before accepting that, ask whether the two numbers average surprise over the same kind of event. A tokenizer is part of a model’s learned interface: changing it changes the token sequence, embedding indices, and probabilities being scored. Per-token perplexities from different model-tokenizer pairs therefore need not be directly rankable.

Information theory turns probability into a measure of surprise. An event of probability $p$ carries self-information $-\log p$: likely events are unsurprising, unlikely ones carry more information when they occur. The base of the logarithm chooses the unit — base two gives bits, the natural logarithm gives nats. Deep-learning libraries almost always use nats, papers frequently quote bits, and the conversion factor $\ln 2 \approx 0.693$ is responsible for a healthy fraction of the wrong numbers in circulation.

The entropy of a discrete distribution is expected surprise:

A distribution concentrated on one outcome has zero entropy. A uniform distribution over many outcomes has the highest entropy available at that size. For a categorical distribution over $V$ candidates, the uniform baseline has perplexity $V$ and surprise $\log_2 V$ bits per event. Apply that anchor only after stating which candidates the runtime includes; Qwen’s tokenizer maps 248,077 ids while its output tensor has 248,320 rows.

To score a model distribution $q$ against outcomes drawn from $p$, use cross-entropy:

On a real corpus the observed next token stands in for the data distribution, so this reduces to averaging the negative log probability the model gave to each token that actually appeared. That is mean negative log-likelihood, the ordinary training loss. Assign tiny probability to what happened and pay heavily; assign high probability and pay little. Cross-entropy decomposes as $H(p,q) = H(p) + D_{KL}(p \Vert q)$: irreducible uncertainty in the data, plus the extra cost of using the wrong distribution. No model can drive the first term to zero, which is why a perplexity of one on natural text would be evidence of leakage rather than genius.

Perplexity exponentiates the average. In nats, $\operatorname{PPL} = \exp(H)$; in bits, $\operatorname{PPL} = 2^{H}$. If the model faced $k$ equally likely options at every step, perplexity would be $k$, which is where the phrase effective branching factor comes from. Real distributions are not uniform, so treat it as an intuition rather than a count of candidates.

For pair A, let the total negative log-likelihood be $S_A=2886$ bits over $N_A=1000$ tokens:

Pair B may produce $N_B=1400$ tokens over the same displayed text, but it has its own total surprise $S_B$:

You cannot substitute $S_A=2886$ into the second formula: a different tokenizer defines different prediction events, and arbitrary ids no longer address the rows learned for them. Holding 2886 fixed would be a useful arithmetic demonstration of denominator sensitivity—yielding about 4.2—but not a valid cross-tokenizer model evaluation. Real comparisons measure both numerators and use a common unit.

The classic wrong turn here deserves naming because it is subtle. Faced with the mismatch, people often try to fix it by comparing per-word perplexity instead, reasoning that words are language-neutral. They are not, as lesson 1.2 showed: word boundaries are a convention that several major languages do not use, and inflected languages pack more into one word than analytic ones. Bytes are not perfect either — a script whose characters cost three UTF-8 bytes is charged more per character than one whose characters cost one — but bytes are at least defined identically for every model and every language, and the definition is checkable.

Comparability has further boundaries beyond tokenization. Perplexity depends on the evaluation corpus, on how beginning and end tokens are handled, on the context length, on whether a sliding window recomputes overlapping regions, and on which positions are masked out of the loss. Two published numbers that differ in any of those are not rankable, and the honest response to a mismatch is to report the discrepancy rather than to average it away.

Leakage is the other quiet destroyer. Evaluation text duplicated in training measures memory alongside generalization, and at modern corpus scales assuming a clean split is optimistic. Domain mismatch cuts both ways: a strong model can look weak on text unlike anything it will serve, or flatteringly strong on text that resembles its training distribution.

Report perplexity the way lesson 1.6 asked you to report an analogy result: with the corpus provenance, the token count, the model and tokenizer versions, the context policy, and enough precision to support the conclusion without exaggerating it. A number without its protocol is not a measurement.