Gradient descent and the loss landscape

1

Concept

Training defines a loss $L(\theta)$ over the model’s parameter vector $\theta$. If there are millions or billions of parameters, the graph of this function lives in an impossibly high-dimensional space. Calling it a loss landscape is useful as long as we remember that every familiar valley drawing is a two-dimensional slice, projection, or toy function.

Basic gradient descent computes $g=\nabla L(\theta)$ and updates $\theta\leftarrow\theta-\eta g$. The negative gradient is the steepest local descent direction under Euclidean distance, and learning rate $\eta$ sets the step. The direction is local: curvature makes the linear approximation less accurate as the step grows. Decreasing loss is therefore not guaranteed for an arbitrary learning rate.

Full-batch gradient descent averages loss over the complete training set before every update. That direction is exact for the finite dataset but expensive. Stochastic gradient descent uses one example. In modern practice, “SGD” often means a minibatch: enough examples for efficient matrix operations and a useful gradient estimate, but far fewer than the dataset. Sampling, shuffling, and distributed batch construction become part of the algorithm.

Minibatch gradients are noisy estimates of the dataset gradient. Larger batches reduce sampling variance but cost more per update and can change optimization and generalization behavior. Doubling batch size does not automatically justify doubling the learning rate across all regimes. Effective batch size also includes gradient accumulation and data-parallel workers. Comparisons should match examples processed, updates, and compute, not merely epochs.

Curvature explains difficult ravines. In a steep direction, a moderate step overshoots from wall to wall; in a shallow direction, the same step advances slowly. The Hessian describes local second derivatives, but forming it explicitly is infeasible at large scale. Momentum and adaptive optimizers approximate useful geometric behavior from gradient history without solving the full second-order problem.

A zero gradient is not enough to identify a solution. It can occur at a local minimum, maximum, saddle point, or flat plateau. High-dimensional neural landscapes contain many flat and saddle-like directions. Parameter symmetries also create equivalent solutions: permuting certain hidden units can leave the function unchanged. Comparing two parameter vectors by straight-line distance may therefore say little about their behaviors.

Training curves are more informative than a beautiful surface alone. Plot training and validation loss against both updates and tokens or examples processed. Watch gradient norms, learning rate, overflow events, and throughput. A jagged batch loss can coexist with a healthy downward trend; a smooth training curve can coexist with overfitting. Repeated seeds reveal whether an apparent improvement exceeds optimization variance.

The landscape metaphor becomes operational when tied to evidence. Gradient gives a local direction, learning rate chooses reach, minibatches provide noisy observations, and curvature determines stability. Optimization does not “understand” the task; it follows the scalar objective provided. A well-designed model and loss reshape the landscape so useful solutions are accessible, while evaluation determines whether descending training loss produced the behavior that actually matters.

When an experiment diverges, preserve the first bad step. Record the batch identity, loss scale, gradient norm, optimizer state, and parameter statistics immediately before and after it. The earliest non-finite value identifies the failing operation far better than the cascade of NaNs seen many updates later.