Scaling laws III: inference-aware & over-training

01 · Concept

Concept

Training-compute optimality asks how to get the lowest loss out of a fixed pretraining budget. A deployed model has a much longer life than that question imagines. Training happens once. Inference happens for every prompt and every generated token, on every replica, for as long as the checkpoint is served. If demand is large, the recurring bill dwarfs the one-time one, and the optimization target has to include both.

The standard framing writes lifetime cost as

with $R$ the expected inference volume and $L$ capturing prompt and output lengths. Sardana and collaborators modified Chinchilla-style analysis along exactly these lines, trained models at high token-to-parameter ratios to validate the behaviour, and reported that sufficiently large inference demand shifts the preferred solution toward smaller models trained on more data. Their crossover point is conditional on their costs, fits, and workload assumptions — but the direction is structural, not incidental.

That is the version of inference-aware scaling everyone quotes, and it treats $N$ as the only lever: pick fewer parameters, feed more tokens, accept a bigger training bill in exchange for a cheaper forward pass forever. Lesson 5.5 walked through that arithmetic. It is real, and it is half the story.

The half that gets skipped is that $C_{infer}$ is not a function of parameter count alone. At a fixed 27 billion parameters you can build architectures whose serving costs differ by more than an order of magnitude, because a served request pays for three things: reading the weights, computing the layers, and holding the per-request state. Only the first scales cleanly with $N$. The third — the KV cache — depends entirely on decisions about heads and layer types, and it is where Qwen3.8-27B makes its position obvious.

Lesson 7.2 owns the KV-cache arithmetic for this model, so this lesson uses its result instead of re-deriving a second budget. The relevant counterfactual holds the head dimension and layer count fixed but replaces four KV heads with 24 and replaces 16 caching attention layers with 64. GQA reduces the KV-head count by a factor of six; the 3

The same distinction applies to the rest of the design. Only 16 layers maintain a cache that grows with sequence length, and the released weights are bf16. Quantized size, usable context, throughput, and total accelerator fit depend on the exact runtime and memory budget; architecture alone does not establish a workstation target or the team’s intent.

Honesty about what is and is not known matters here as much as anywhere in this track. The architectural facts above are verifiable from the model card and config. The reasoning attributed to them is inference from the design, not a published rationale — there is no technical report for this model, and the team never stated their serving assumptions, token budget, or cost model. What you can say with confidence is that the shipped architecture reduces growing per-sequence state relative to the stated counterfactual. Interpreting that effect as a team objective is plausible but remains inference.

The usual caveats still bind. Forecasts of $R$ are frequently wrong in both directions, so report sensitivity across demand scenarios rather than a single point estimate. Matching validation loss does not guarantee matching latency-sensitive quality, factuality, multilingual behaviour, or post-training responsiveness. Data availability constrains the over-training lever: extra unique high-quality tokens may simply not exist, and repeating data changes the effective objective and raises memorization risk. Update cadence matters too — a checkpoint replaced next month accumulates far less lifetime inference than one served for years.

The durable principle is to optimize the system’s lifetime rather than a single run. Chinchilla asks how to spend a pretraining budget. Inference-aware scaling asks what should exist after training, given how often it will be used — and the answer is expressed in tokens, in parameters, and, most consequentially, in the shape of the layers themselves.