Gradient descent and the loss landscape

01 · Concept

Concept

Backpropagation hands you a direction. It does not tell you how far to walk. That single missing number — the step size — is responsible for more failed training runs than any architectural choice, and understanding why requires looking at the shape of the function being descended rather than at the algorithm.

Training defines a loss $L(\theta)$ over the parameter vector $\theta$. Basic gradient descent computes $g=\nabla L(\theta)$ and updates

where the negative gradient is the steepest descent direction under Euclidean distance and $\eta$ is the learning rate. The direction is strictly local; the linear approximation it comes from degrades as the step grows, so nothing guarantees the loss decreases for an arbitrary $\eta$.

Watch the problem appear in a function you can solve on paper. Let

a bowl that is a hundred times steeper along $x$ than along $y$. The gradient is $(10x,\,0.1y)$, so one step of descent gives $x\leftarrow(1-10\eta)x$ and $y\leftarrow(1-0.1\eta)y$. Each coordinate is multiplied by its own factor every step, and the whole story is in those two factors.

Try $\eta=0.15$. The $x$ factor is $1-1.5=-0.5$: the sign flips every step, so the iterate bounces across the valley floor, though the magnitude halves each time and it does converge. The $y$ factor is $1-0.015=0.985$: after a hundred steps $y$ has shrunk only to $0.985^{100}\approx0.22$, and reaching one percent of its starting value takes over three hundred steps. So the shallow direction crawls.

The natural reaction is the classic wrong turn: progress along $y$ is too slow, therefore raise the learning rate. Try $\eta=0.25$. Now the $y$ factor improves to $0.975$, which is what you wanted. But the $x$ factor becomes $1-2.5=-1.5$, whose magnitude exceeds one, so $x$ grows by half every step with alternating sign and the run diverges. The divergence threshold here is exactly $\eta=0.2$, set entirely by the steep direction, while the shallow direction would happily accept a rate a hundred times larger. One global step size cannot serve both, and the gap between them is the curvature ratio — a hundred in this toy, and vastly larger in a real network.

Real training adds a second complication: you never see the true gradient. Full-batch descent averages the loss over the complete training set before each update — exact for that dataset, and unaffordable. Stochastic gradient descent uses one example. Modern practice sits between: a minibatch large enough for efficient matrix operations and a usable estimate, far smaller than the dataset. Sampling, shuffling, and distributed batch assembly become part of the algorithm rather than plumbing around it.

Minibatch gradients are noisy estimates. Larger batches reduce sampling variance but cost more per update, and they change optimization and generalization behavior in ways that do not follow a simple rule — doubling the batch does not license doubling the learning rate across all regimes. Effective batch size also includes gradient accumulation steps and data-parallel workers, so two runs described as “batch 512” may differ. Comparisons should match examples processed, update count, and compute, not merely epochs.

Now the tie to scale. For Qwen3.8-27B, $\theta$ is a vector with roughly 27 billion coordinates (Qwen3.8-27B Model Card, 2026). Every picture you have seen of a loss landscape — the smooth bowl, the dramatic ridged surface — is a two-dimensional slice or projection through a space of that dimension, chosen after the fact and usually along directions that make the story legible. Such plots are honest illustrations and dishonest evidence. Nothing about a 27-billion-dimensional geometry can be read off a surface rendered in three.

What you can trust is instrumentation. Plot training and validation loss against updates and against tokens or examples processed, since those axes disagree whenever batch size changes. Watch gradient norms, the learning-rate schedule, overflow events in reduced precision, and throughput. A jagged per-batch loss can coexist with a healthy trend; a beautifully smooth training curve can coexist with overfitting. Repeated seeds tell you whether an apparent improvement clears optimization variance.

And when a run does diverge, preserve the first bad step rather than the wreckage. Record the batch identity, loss scale, gradient norm, optimizer state, and parameter statistics immediately before and after it. The earliest non-finite value names the failing operation; the cascade of not-a-numbers observed many updates later names nothing at all.