Optimizers: SGD to Momentum to Adam to AdamW

01 · Concept

Concept

Lesson 2.6 left a concrete failure on the table. In a bowl a hundred times steeper along one axis than another, any learning rate small enough to keep the steep direction stable leaves the shallow direction crawling, and any rate fast enough for the shallow direction blows the steep one up. Backpropagation cannot help — it reported both gradients correctly. The fix has to live in how the update uses them, which is what an optimizer is.

Plain minibatch SGD does the minimum: $\theta_t=\theta_{t-1}-\eta g_t$. Little state, clear interpretation, and exactly the failure described above.

Momentum adds a velocity, usually an exponential moving average of gradients, and steps along that instead of the raw gradient. Components that persist across steps accumulate; components that alternate in sign partly cancel. In the ravine, the wall-to-wall bouncing along the steep axis largely cancels itself while the steady push along the shallow axis builds up. Different libraries use slightly different equations and Nesterov variants, so hyperparameters do not transfer by name alone.

Adam attacks the scale mismatch directly. Per coordinate it keeps a first moment $m_t$, a smoothed gradient, and a second raw moment $v_t$, a smoothed squared gradient. After bias correction — necessary because both averages start at zero and are otherwise biased toward it early on — the update is roughly

Work out what that does to the ravine. Along the steep axis the gradients are consistently large, so $\sqrt{\hat v}$ is large and divides the step back down. Along the shallow axis the gradients are consistently tiny, so $\sqrt{\hat v}$ is tiny and divides the step back up. If a coordinate’s gradient has a steady magnitude $|g|$, then $\hat m\approx g$ and $\sqrt{\hat v}\approx|g|$, so the ratio is approximately $\pm1$ and the parameter moves by about $\eta$ — regardless of whether its gradient was $10x$ or $0.1y$. Adam has made the step size roughly scale-free per direction. That is the property lesson 2.6 said one global $\eta$ could not provide.

Now the classic wrong turn, which shipped in real code for years. You want regularization, so you add an L2 penalty to the loss. Its gradient contribution is $\lambda\theta$, and you hand the sum to Adam. Reason about what happens next: that term goes into $m$ and $v$ along with everything else, and then the whole thing is divided by $\sqrt{\hat v}$. A parameter whose gradients happen to be large gets its decay shrunk; a parameter with tiny gradients gets its decay amplified. The intended behavior — pull every weight toward zero by the same proportion each step — is not what occurs, and the strength of the regularizer now depends on unrelated gradient statistics. AdamW fixes it by decoupling: compute the adaptive update, apply it, then separately shrink the parameters by $\eta\lambda\theta$. Same name, different operation, and the reason essentially every large model today is trained with AdamW rather than Adam plus L2.

Not every parameter should decay. Biases and normalization scale parameters are commonly placed in a no-decay group, a configuration detail that is a frequent source of silent bugs — log which parameter names entered which group and fail loudly on unexpected omissions.

Schedules matter as much as the update rule. Warmup keeps early effective steps small while activations and moment estimates settle; decay reduces the step later for convergence. A fair experiment reports the optimizer equations or implementation, the learning-rate curve, batch size, clipping threshold, decay groups, total tokens, and seeds. SGD, Momentum, Adam, and AdamW are update rules with different inductive biases, not a chronological ladder on which the newest name wins.

One operational detail that costs people days: optimizer checkpoints must include the step counter and both moment buffers, not only the model weights. Resuming Adam with empty moments silently changes the update rule for hundreds of steps even though the parameters match exactly, and stepping the schedule once per batch rather than once per accumulation boundary produces a different learning-rate curve entirely. A reproducible run records those boundaries and verifies the first resumed update against an uninterrupted reference.