Scaling laws I: Kaplan

01 · Concept

Concept

You have budget for exactly one large training run and four candidate configurations. Training all four to find out which was sensible is not an option; the run is the budget. So you do what shipbuilders do with towing tanks: you run a family of small, controlled experiments, fit a curve through them, and use the curve to choose. Scaling laws are that curve. The remarkable empirical finding behind them is that held-out loss falls along strikingly smooth trends as parameters, data, and compute grow, even though the underlying networks are anything but smooth.

The standard functional form is a power law plus a floor:

Here $x$ is non-embedding parameter count, dataset size, or training compute. $L_\infty$ is the irreducible term — the entropy the data itself carries, which no model removes. $A x^{-\alpha}$ is the reducible part, and on log-log axes it becomes a straight line whose slope is $-\alpha$. A larger positive $\alpha$ means loss falls faster as that resource grows.

Fit it once by hand, because doing so exposes the trap. Suppose two controlled runs share a corpus, tokenizer, and recipe and differ only in size: 100 million non-embedding parameters reaches held-out loss 3.20, and 1 billion reaches 2.80. Pilot work estimates the floor at $L_\infty = 1.70$. Subtract it before doing anything else:

Now predict a third point. At 10 billion parameters the reducible part should be $1.10\times 0.7333 = 0.807$, so $L \approx 1.70 + 0.81 = 2.51$. You can go train that run and check the residual — which is the entire point of a fitted law.

Here is the wrong turn, and it is the one people actually take. Skip the floor and fit the raw losses directly. The ratio is $2.80/3.20 = 0.875$ per decade, giving $\alpha = -\log_{10}(0.875) = 0.058$. Over the two fitted points this looks perfectly respectable — it reproduces both exactly. Extrapolate four decades to 10 trillion parameters and it predicts

a loss below the irreducible floor of 1.70. The naive fit has forecast a model that predicts the data better than the data’s own entropy allows. Nothing in the arithmetic complains; the impossibility only shows up when you know a floor exists and check against it. Always fit $L_\infty$ jointly, and always sanity-check the extrapolation against it.

Kaplan and collaborators trained many autoregressive Transformer language models and studied loss against model size, data, and compute. Their central structural claim is that performance is limited by whichever resource is too small: parameters cannot indefinitely compensate for insufficient data, and tokens cannot indefinitely compensate for a tiny model. For a fixed compute budget you must therefore choose a point on a tradeoff — a larger model spends more operations per token and so sees fewer tokens; a smaller one sees more tokens with less capacity. The Kaplan-era fitted optimum favoured growing model size relatively fast and stopping training comparatively early, and that recommendation shaped early large-model planning until the next lesson overturned it.

The compute axis usually uses the first-order estimate $C\approx 6ND$ floating-point operations for $N$ non-embedding parameters and $D$ tokens, with the constant depending on what is counted and on architecture. Lesson 5.13 applies it in anger. Note that FLOPs are not wall-clock time: utilization, communication, data loading, recomputation, and failures decide elapsed cost.

That caveat has teeth for the specimen this course follows. Qwen3.8-27B publishes no scaling study, no token count, and no training compute — there is no technical report at all, only the model card and config. So you cannot place it on anyone’s fitted curve. Worse, you should not assume a curve fitted on uniform transformers transfers to it, because its architecture is not uniform: 48 of its 64 layers are Gated DeltaNet rather than full attention (lesson 4.16), which changes the FLOPs-per-token constant and the way cost grows with sequence length. The $6ND$ shorthand and the exponents that accompany it were measured on stacks where every layer was the same. A hybrid stack is a different tank, and the hull curves have to be refitted.

Good scaling work separates interpolation from extrapolation. Fit on some runs, hold out others, and check prediction error before trusting a curve beyond the measured range. Plot residuals rather than admiring a straight line: a small systematic deviation becomes expensive across orders of magnitude. Loss is also not the product. Two models with matching validation loss can differ in downstream tasks, calibration, safety, multilingual coverage, latency, and memory, and benchmark thresholds that look like emergence often mark a smooth capability crossing a scoring boundary.

The $D$ axis carries its own complication, straight out of lesson 5.2: a token is not a standard unit of information. Duplicated boilerplate, low-quality web text, and curated domain data all count as tokens and contribute differently, so effective data size depends on duplication and mixture. Hold the pipeline fixed across a scaling study, or model the pipeline change explicitly, or the exponent you fit is measuring your filters.

The workflow, then, is disciplined rather than clever: choose several sizes and token budgets, keep recipes comparable, reserve validation data, log realized compute, fit competing functional forms, validate on withheld runs, quantify uncertainty, and refit whenever data or architecture changes. What endures is predictability with boundaries — the curve makes enormous budget decisions empirical instead of superstitious, and the humility to say where it stops applying is part of the method, not an apology for it.