Scaling laws II: Chinchilla & compute-optimality

01 · Concept

Concept

A colleague sees that Qwen3.8-27B has 27 billion parameters, remembers the famous “twenty tokens per parameter” rule, multiplies, and announces that the model was trained on about 540 billion tokens. It is a natural inference, the arithmetic is right, and the conclusion is unsupported by the published evidence. Understanding why requires knowing exactly what Chinchilla measured, what it optimized, and what has changed since.

Kaplan-style laws made planning predictable, but their compute-optimal allocation was not the last word. In 2022 Hoffmann and collaborators revisited the question with a broader set of iso-compute experiments and produced the model called Chinchilla. The interesting output was not a checkpoint. It was a different answer to “how large should the model be, and how many tokens should it see?”

For a fixed training budget $C$, a dense model faces the approximate constraint $C\propto ND$, with $N$ parameters and $D$ tokens. An iso-compute curve fixes $C$, picks several values of $N$, trains each for the corresponding $D$, and compares validation loss. Too large a model starves for data; too small a model lacks capacity despite abundant tokens. The bottom of that U-shaped curve is the best balance at that budget. Repeat across several budgets, fit how the optimal $N$ and $D$ grow together, and you have a rule. Hoffmann et al. concluded that in their measured setting, parameters and tokens should scale at roughly equal rates. Their headline comparison trained a 70-billion-parameter Chinchilla on 1.4 trillion tokens against the larger 280-billion-parameter Gopher on 300 billion tokens, at the same stated training compute, and Chinchilla won on the reported suite. That ratio, 1.4 trillion over 70 billion, is where “about twenty tokens per parameter” comes from.

Now do the arithmetic your colleague did, but carefully. At twenty tokens per parameter, a 27-billion-parameter dense model would be trained on

and the training compute, using the $6ND$ estimate from lesson 5.4, would be

That is a real number and a useful anchor: it tells you the order of magnitude of a compute-optimal 27B run. It is also entirely our construction. No token count, compute figure, or training budget is published for Qwen3.8-27B. There is no technical report; the model card documents architecture, licence, sampling presets, and vendor-reported benchmarks. So treat 8.75e22 FLOPs as “what a Chinchilla-optimal 27B would have cost”, never as “what this model cost”.

Two refinements before moving on. The $N$ in these fits is non-embedding parameters, and lesson 5.3 showed that roughly 2.54 billion of this model’s parameters live in the embedding and untied output head, with more in the vision tower — so a strict application would use something closer to 24 billion and shrink the estimate by around 10%. And $6ND$ was calibrated on uniform attention stacks, whereas 48 of this model’s 64 layers are Gated DeltaNet. Both corrections point the same way: these are order-of-magnitude planning figures, and quoting them to three significant digits is false precision.

The conceptual correction Chinchilla delivered was undertraining. A parameter earns its keep through many varied token contexts; building more parameters while starving each of data is worse than training a smaller network longer. After 2022, “tokens per parameter” became planning shorthand — but a single ratio cannot capture data quality, architecture, or a shifted loss regime.

That something is deployment. Chinchilla’s question ends the moment training stops. If a model will serve billions of requests, a smaller model trained longer costs more once and less forever. Push our 27B past the compute-optimal point to an illustrative 150 tokens per parameter — again, our assumption, not a published fact — and the budget becomes

roughly 7.5 times the Chinchilla-optimal budget for the same parameter count. From a training-only viewpoint that is waste. From a serving viewpoint it is a purchase: those extra tokens push a 27-billion-parameter model toward quality that would otherwise have required a much larger one, and the smaller model then reads 54 GB of weights per forward pass instead of several hundred, for every token of every request, for the life of the deployment. Lesson 5.6 turns that trade into an explicit objective and separately examines the serving consequences of the published Qwen3.8-27B architecture, without claiming that the team trained it under this token-budget rationale.

The methodological lesson outlasts the numbers. A famous scaling recommendation changed because experimental coverage and fitting improved, and it will change again. Plan a large run from controlled pilot evidence, attach uncertainty, refit when the pipeline or architecture moves — and remember that compute-optimality answers the question it was asked. “Bigger model” and “more training” compete for the same training budget. Neither of them decides the lifetime economics of a deployed system, which is where this track goes next.