Scaling laws I: Kaplan

1

Concept

Training a frontier model just to discover whether a configuration was sensible is prohibitively expensive. Scaling laws use families of smaller experiments to estimate how held-out loss changes as parameters, data, and compute grow. The striking observation is that these relationships can be smooth over substantial ranges even though the networks are complex.

A common form is a power law plus an irreducible term:

$x$ might be non-embedding parameter count, dataset size, or training compute. On log-log axes, the reducible part becomes approximately linear, making the exponent $\alpha$ visible as a slope. Bigger $\alpha$ means loss falls faster as that resource grows. The fitted curve summarizes observed runs; it does not explain every mechanism producing the trend.

Kaplan and collaborators trained many autoregressive Transformer language models and studied loss as a function of model size, data, and compute. Their analysis emphasized that performance was limited by whichever resource was too small: adding parameters cannot compensate indefinitely for insufficient data, and adding tokens cannot compensate indefinitely for a tiny model.

For a fixed compute budget, one must choose a point on a tradeoff. A larger model spends more operations per token and therefore sees fewer tokens. A smaller model processes more tokens but has less parameter capacity. The Kaplan-era fitted optimum favored scaling model size relatively quickly and stopping training before many passes over the available distribution. That recommendation influenced early large-model planning.

Compute for a dense Transformer is often approximated to first order as proportional to parameters times training tokens. A frequently used training estimate is $C\approx6ND$ floating-point operations for $N$ non-embedding parameters and $D$ tokens, but the constant depends on what operations are counted and on architecture. FLOPs are not wall-clock time: hardware utilization, communication, data loading, recomputation, and failures determine elapsed cost.

Good scaling work separates interpolation from extrapolation. Fit on some runs, hold out others, and check prediction error before trusting a curve beyond the measured range. Plot residuals rather than only a beautiful straight line. A small systematic deviation can become expensive when extrapolated across orders of magnitude.

Loss also is not the entire product. Two models with similar validation loss can differ on downstream tasks, calibration, safety, multilingual coverage, latency, and memory. Emergent-looking benchmark thresholds can arise when a smooth underlying capability crosses a task’s scoring boundary. A scaling curve guides allocation; it does not replace evaluation.

Data quality complicates the $D$ axis. A token is not a standard unit of information. Repeated boilerplate, low-quality text, and valuable domain data all count as tokens while contributing differently. Effective data size depends on duplication and mixture. Scaling experiments must hold the pipeline stable or model a pipeline change explicitly.

The practical workflow is disciplined: choose several model sizes and token budgets; keep training recipes comparable; reserve validation data; log realized compute; fit competing functional forms; validate on withheld runs; quantify uncertainty; then use the curve to select the next experiment. Refit when architecture or data changes.

The lasting insight is predictability with boundaries. Language-model loss can follow smooth resource trends, making empirical science useful for enormous budget decisions. The equally important lesson is humility: the curve describes a regime. The next lesson shows how a new experimental design changed the recommended parameter-to-token allocation.