Derivatives and gradients, geometrically

1

Concept

A derivative measures local sensitivity. For a one-variable function $f(x)$, the derivative at $x$ is the slope approached by increasingly small secant lines:

If $f'(x)=3$, a sufficiently small positive change $\Delta x$ produces an output change of roughly $3\Delta x$. This is a local linear approximation, not a promise about large moves or distant regions.

Geometrically, the derivative is the slope of the tangent line. A positive value rises as $x$ increases; a negative value falls; zero means locally flat, though not necessarily a minimum. The point might be a maximum, a minimum, or a flat inflection. Derivatives describe nearby behavior. They do not automatically reveal the best point over an entire function.

Neural networks depend on many parameters, so their loss is a function $L(\theta_1,\ldots,\theta_n)$. A partial derivative $\partial L/\partial\theta_i$ asks how loss changes when parameter $i$ moves slightly while the others are held fixed. Collecting all partial derivatives gives the gradient:

The gradient points in the direction of steepest local increase under Euclidean distance. Its negative points toward steepest local decrease. Gradient descent uses the update $\theta\leftarrow\theta-\eta\nabla L$, where the learning rate $\eta$ sets the step size. Too small can make progress slow; too large can overshoot a valley or become unstable. Curvature means the useful step can vary by direction and over time.

The chain rule connects local sensitivities through a composition. If $y=f(u)$ and $u=g(x)$, then $dy/dx=(dy/du)(du/dx)$. A neural network is a long composition of matrix multiplications, activations, normalization, and loss computation. Backpropagation applies the chain rule efficiently, reusing intermediate derivatives instead of separately expanding every path from every parameter to the loss.

Gradients have units and scale. Multiplying the loss by one thousand multiplies its gradient by one thousand without changing which parameters represent a solution. Features with very different scales can make the surface poorly conditioned: one direction may be steep while another is shallow. Normalization, initialization, adaptive optimizers, and learning-rate schedules all help manage the geometry, but none changes the basic meaning of the derivative.

Not every operation is differentiable everywhere. ReLU has a corner at zero; software chooses a conventional subgradient there. Token selection is discrete, so training differentiates through probabilities and continuous representations rather than through a sampled word choice. Numerical finite differences can check a small implementation by perturbing a parameter, but analytic automatic differentiation is far cheaper for millions or billions of parameters.

The crucial mental model is a local map. Loss is terrain over parameter space. A partial derivative is a slope along one coordinate; the gradient is the combined uphill arrow; the negative gradient suggests a downhill move. Optimization is difficult not because the definition is mysterious, but because the terrain is enormous, noisy, curved, and observed through batches. Geometry explains both why gradient methods work and why one local arrow cannot guarantee the global destination.