Optimizers: SGD to Momentum to Adam to AdamW

1

Concept

Backpropagation produces gradients; an optimizer turns them into parameter updates. The optimizer adds state and policy to the local signal. Its behavior depends on learning rate, schedule, batch construction, clipping, precision, and regularization. Comparing names without matching those conditions is rarely informative.

Plain minibatch SGD updates $\theta_t=\theta_{t-1}-\eta g_t$. It has little state and a clear interpretation. Gradient noise can help exploration, but narrow curved valleys cause oscillation. Tuning a single global learning rate is difficult when useful scales differ across parameters.

Momentum maintains velocity, commonly an exponential moving average of gradients. Persistent directions accumulate speed, while alternating components partly cancel. This can move quickly along a shallow valley and reduce wall-to-wall bouncing. Different libraries use slightly different equations or Nesterov variants, so hyperparameters are not perfectly portable by label alone.

Adam estimates a first moment $m_t$ and second raw moment $v_t$ for each parameter coordinate. Roughly, the update divides the bias-corrected first moment by the square root of the bias-corrected second moment plus a small $\epsilon$. Recent gradient direction drives the step; recent squared magnitude normalizes it. Bias correction matters early because moving averages initialized at zero are otherwise biased toward zero.

Adaptive scaling is useful for sparse or uneven gradients, and Adam became common in Transformer training. It is not automatically stable under every setting. The meaning and placement of epsilon, precision of optimizer state, gradient clipping, and beta values matter. Optimizer state also consumes memory: Adam normally stores two moment tensors in addition to parameters and gradients.

Regularization creates a subtle distinction. For ordinary SGD, adding an L2 penalty to the objective can be equivalent to shrinking weights during updates under standard conditions. In Adam, the penalty gradient is passed through coordinate-wise adaptive scaling, so it no longer acts as uniform shrinkage. AdamW decouples weight decay: first compute the adaptive gradient update, then apply explicit parameter decay. This makes the decay behavior match its intended definition more directly.

Not every parameter should necessarily decay. Biases and normalization scale parameters are often placed in no-decay groups, depending on architecture and evidence. Parameter grouping is part of the optimizer configuration and a frequent source of silent bugs. Log which names enter each group and fail on unexpected omissions or duplicates.

Schedules matter as much as the optimizer. Warmup avoids large effective updates before activations and moment estimates settle. Decay reduces step size later for convergence. A fair experiment reports optimizer equations or implementation, learning-rate curve, batch size, clipping, decay groups, total tokens, and seeds. SGD, Momentum, Adam, and AdamW are update rules with different inductive biases—not a chronological ladder where the newest name wins every task.

Optimizer checkpoints must include step counters and moment buffers, not only model weights. Resuming Adam with empty moments changes the update rule even when parameters match exactly. Likewise, stepping the schedule once per batch versus once per epoch can produce a radically different learning-rate curve. A reproducible run records those boundaries and verifies the first resumed update against an uninterrupted reference.