Scaling laws II: Chinchilla & compute-optimality

1

Concept

Kaplan-style laws made language-model planning more predictable, but their compute-optimal allocation was not the final word. In 2022, Hoffmann and collaborators revisited the question with a broader set of iso-compute experiments and produced the model known as Chinchilla. The important result was not simply a better checkpoint. It was a different answer to “how large should the model be, and how many tokens should it see?”

For a fixed training budget $C$, a dense model faces an approximate constraint $C\propto ND$, where $N$ is parameters and $D$ is tokens. An iso-compute curve chooses several values of $N$, trains each for the corresponding $D$, and compares validation loss. Too large a model sees too little data. Too small a model lacks capacity despite seeing many tokens. The bottom of the curve estimates the best balance for that budget.

Repeat this process across several budgets and fit how the optimal $N$ and $D$ grow. Hoffmann et al. concluded that, in their measured setting, model size and training tokens should scale at roughly equal rates as compute increases. Their representative result compared a 70-billion-parameter Chinchilla model trained on 1.4 trillion tokens with the larger 280-billion-parameter Gopher model trained on 300 billion tokens; those figures are reported in the 2022 paper. Chinchilla used the same stated training compute while outperforming Gopher on the reported suite.

The conceptual correction was undertraining. A parameter receives useful pressure through many varied token contexts. Building more parameters while starving each of data can be worse than training a smaller network longer. After Chinchilla, “tokens per parameter” became a common planning shorthand, though one ratio cannot capture data quality, architecture, or changing loss regimes.

Compute-optimality also depends on what cost is optimized. Chinchilla’s core question concerns a fixed pretraining compute budget and model loss. If a model will serve billions of requests, a smaller model trained longer may cost more to train but far less to operate. If training data is scarce, duplicated tokens are not equivalent to new information. If latency requires a particular width or depth, architecture constraints change the feasible set.

Reproducing the method requires more than fitting a line to public checkpoints. Training recipes must be tuned across sizes; otherwise a small model with a bad learning rate appears artificially weak. Token and parameter definitions must be consistent. Runs should approach their intended data budget without divergence. The validation set must be held out and deduplicated from training.

An interactive log-log chart should show a family of U-shaped iso-compute curves, not only the final scaling line. Moving right increases parameters and reduces tokens under fixed compute. Each curve’s minimum identifies a balance. Connecting minima across budgets gives the fitted scaling rule. This makes the conclusion visibly empirical.

The Chinchilla lesson is methodological as much as numerical. A famous scaling recommendation can change when experimental coverage and fitting improve. Plan a large run from controlled pilot evidence, attach uncertainty, and refit when the data pipeline or architecture changes.

The durable takeaway is that “bigger model” and “more training” compete for the same compute. Compute-optimality finds a balance for a stated objective. It does not decide the lifetime economics of deployment, which introduces inference demand in the next lesson.