Scaling laws III: inference-aware & over-training

1

Concept

Training-compute optimality asks how to obtain the lowest loss from a fixed pretraining budget. A deployed model has a longer life. Training happens once; inference happens for every prompt and generated token. If demand is large, recurring cost can dominate the one-time training bill. The optimization target should then include both.

Suppose two models reach similar quality. Model A is larger and trained for fewer tokens. Model B is smaller and trained longer. A training-only analysis may prefer A because the final increments of data for B are relatively inefficient during pretraining. A lifetime analysis may prefer B because every future forward pass touches fewer parameters and requires less memory bandwidth.

This is called over-training only relative to a training-compute optimum such as a fitted Chinchilla point. It does not mean training past useful learning or overfitting the validation distribution. The additional tokens purchase a smaller deployed model at a target loss. The name is potentially misleading unless the reference optimum is stated.

A simple objective writes total cost as

where $R$ is expected inference volume and $L$ captures prompt and output lengths. Real systems should separate prefill from decode, batch size, cache reuse, hardware type, energy, memory capacity, and utilization. Parameter count is a useful proxy, not a complete serving model.

Sardana and collaborators modified Chinchilla-style analysis to include inference and trained a family of models to validate behavior at high token-to-parameter ratios. Their 2024 paper reports that sufficiently large inference demand shifts the preferred solution toward smaller models trained on more data. The exact crossover is conditional on their costs, fits, and workload assumptions.

Quality is also multidimensional. Matching validation loss does not guarantee equal latency-sensitive task quality, factuality, multilingual behavior, or post-training response. A smaller heavily trained model can be attractive, but benchmark and safety evaluation must verify the intended target. Architecture shape matters too: models with the same parameter count can have different latency because depth limits sequential execution and width changes matrix efficiency.

Data availability creates a hard constraint. Extra unique high-quality tokens may not exist. Repeating data changes the effective objective and can increase memorization. Synthetic data has generation cost and quality dependencies. An inference-aware plan should model the marginal value of later tokens, not assume all $D$ is equally informative.

The choice also interacts with update cadence. If a checkpoint will be replaced in one month, lifetime inference volume is smaller than for a model kept for years. Multiple regional replicas, quantization, speculative decoding, and hardware amortization alter serving economics. A licence or data-removal requirement may shorten model life.

A useful calculator exposes assumptions: target loss, candidate parameters, training tokens, accelerator throughput, training utilization, requests, input and output lengths, batching, energy, and capital or rental price. It should show a range, not a magical total, and keep empirical coefficients tied to their source and date.

The lasting principle is to optimize the system’s lifetime, not a single training run. Chinchilla asks how to spend a pretraining budget. Inference-aware scaling asks what model should exist after training, given how often it will be used. More training can be rational when it buys a smaller machine for every future token.