Overfitting, regularization, and the bitter lesson

01 · Concept

Concept

A model reaches a training loss of 0.05 while a colleague’s reaches 0.40. Which is better? The honest answer is that the question is unanswerable as posed, and the reason is the subject of this lesson.

A model generalizes when patterns learned from training data support performance on new examples drawn from the distribution it will actually face. Overfitting is the gap between fitting the observed sample and performing on representative unseen cases. It is not the same thing as having many parameters and it is not diagnosed by a low training loss. A small model can overfit badly; a very large one can generalize well. What settles it is evidence from data the model did not train on.

That evidence has to be structured deliberately. The training set updates parameters. A validation set guides architecture, hyperparameters, stopping point, and thresholds. A test set estimates the finally selected system. The rule that people break is the last one: check the test set repeatedly, adapt to what you see, and it has quietly become a validation set, so the number you report is optimistic by an unknown amount. Worse and more common, near-duplicate documents or temporally leaked examples can corrupt every split while the splitting code remains formally correct.

Learning curves are how the gap becomes visible. If training and validation losses are both high, the model may be underfitting or the optimization may be broken. If training loss falls while validation loss rises, further fitting is now harming performance under that validation distribution. Curves need uncertainty across seeds and a meaningful horizontal axis — tokens or examples processed, not epochs, when batch sizes differ between the runs being compared.

The regularizer catalogue is short and each entry encodes an assumption. Weight decay discourages large parameters, as lesson 2.7 detailed. Dropout randomly masks activations during training so the network cannot rely on one exact co-adaptation. Data augmentation injects label-preserving variation — valid only when the transformation really does preserve the label, since an invalid augmentation teaches a wrong invariance with full confidence. Early stopping picks a checkpoint before validation performance degrades. Label smoothing, parameter sharing, and simply using a smaller model belong on the same list.

At large scale, the dataset itself becomes part of the regularization design. More representative coverage can reduce sampling error, diversity can expose shortcuts that fit only a narrow slice, and deduplication can limit the influence of repeated examples. None of that makes data a universally dominant regularizer or makes dropout, weight decay, and early stopping obsolete. Their relative value depends on the data distribution, objective, architecture, optimization, and evaluation evidence. Treat each as an empirical choice to ablate, not a hierarchy that holds across every training regime.

Modern deep learning also complicates the old story that capacity causes overfitting. Heavily overparameterized networks can interpolate their training data and still generalize, in regimes where implicit biases of the optimizer, the architecture, and the structure of the data do the work that explicit regularization used to. Parameter count on its own is therefore a poor diagnosis. Empirical curves, ablations, contamination checks, and out-of-distribution tests carry the evidence.

The working discipline is unglamorous. Define the population you actually intend to serve before celebrating a training curve. Keep a genuinely held-out path and record every selection decision made against validation. Evaluate the slices where harm or failure concentrates rather than the aggregate alone. Use regularization to encode assumptions you can defend rather than as a bag of switches. Prefer scalable learning methods, while remembering that compute magnifies the consequences of your data and your objective in both directions. Generalization is demonstrated on new evidence and never by the beauty of a loss curve.

One last practical note: when the future distribution drifts over time, add a time-based holdout alongside the random split. It exposes dependence on stale templates and repeated entities that random shuffling hides completely.