Derivatives and gradients, geometrically

01 · Concept

Concept

Here is the problem calculus solves for us. A model has parameters, and some setting of them makes its predictions good. You have no map of that space, no formula for the best setting, and far too many knobs to try combinations. All you can do is evaluate how bad the current setting is and ask a local question: if I nudge this one number slightly, does the badness go up or down, and by how much? Answer that for every number at once and you have a direction to move. That is the entire content of gradient-based learning, and everything below is the machinery for asking the question precisely.

A derivative measures local sensitivity. For a one-variable function $f(x)$, the derivative at $x$ is the slope approached by ever-smaller secant lines:

If $f'(x)=3$, then a sufficiently small change $\Delta x$ produces an output change of roughly $3\Delta x$. That is a local linear approximation, not a promise about large moves. Geometrically it is the slope of the tangent line: positive means rising, negative means falling, zero means locally flat. Flat does not mean minimal; the point could be a maximum, a minimum, or an inflection where the curve merely pauses.

Neural networks depend on many parameters, so the loss is a function $L(\theta_1,\ldots,\theta_n)$. A partial derivative $\partial L/\partial \theta_i$ asks how the loss changes when parameter $i$ moves slightly while all others stay fixed. Collecting them all gives the gradient:

The gradient points toward steepest local increase; its negative points downhill. Gradient descent applies $\theta \leftarrow \theta - \eta \nabla L$, where the learning rate $\eta$ sets the stride.

Work the simplest possible example completely, because the failure mode is more instructive than the success. Let $L(w) = w^2$, so $\nabla L = 2w$, with the obvious minimum at zero. Start at $w = 1$ with $\eta = 0.1$. The first step gives $1 - 0.1 \times 2 = 0.8$. The second gives $0.8 - 0.1 \times 1.6 = 0.64$. Then $0.512$, then $0.4096$. Each step multiplies $w$ by $0.8$, so the sequence marches steadily toward the minimum. Optimization looks easy.

Now change nothing but the learning rate, to $\eta = 1.1$. From $w = 1$: $1 - 1.1 \times 2 = -1.2$. From $-1.2$: $-1.2 - 1.1 \times (-2.4) = 1.44$. Then $-1.728$, then $2.0736$. The magnitude grows by 20 percent every step and the loss explodes. Look carefully at what went wrong, because it was not the calculus. Every gradient in that run was exactly correct, and every step moved in the correct direction. The stride simply carried the parameter clean across the valley and up the far wall, into territory the local reading never described. This is the origin of exploding losses, of learning-rate warmup, and of the schedules in lesson 5.11.

The chain rule connects local sensitivities through composed operations. If $y = f(u)$ and $u = g(x)$, then $dy/dx = (dy/du)(du/dx)$. A neural network is a long composition of matrix multiplications, activations, normalizations, and finally a loss, so its derivative is a long product of local factors. Backpropagation is the efficient bookkeeping for that product: it reuses intermediate results instead of separately tracing every path from every parameter to the loss, which is what makes training a large model arithmetically possible at all. Lesson 2.5 works through the mechanics.

That efficiency matters at the scale this course studies. In a model such as Qwen3.8-27B, the gradient is not a small arrow: it is a vector with one component for every one of roughly 27 billion parameters, the same shape as the parameters themselves, recomputed on every batch. Whatever else training a large model involves, geometrically it is stepping against that arrow, over and over.

Gradients carry units and scale. Multiplying the loss by a thousand multiplies its gradient by a thousand without changing which parameters constitute a solution. Features on wildly different scales make the surface poorly conditioned: one direction is a cliff while another is nearly flat, and a single scalar learning rate cannot suit both. Normalization, careful initialization, adaptive optimizers, and schedules all exist to manage that geometry. None of them changes the meaning of a derivative; they change the terrain it is measured on, which is the subject of lessons 2.6 through 2.8.

Not everything is differentiable everywhere. ReLU has a corner at zero and software picks a conventional value there. Token selection is discrete, so training differentiates through probabilities and continuous representations rather than through a sampled word. Finite differences can sanity-check a tiny implementation by perturbing one parameter and re-evaluating, but analytic automatic differentiation is enormously cheaper once the parameter count passes a few hundred.

The durable picture is a local map. Loss is terrain over parameter space; a partial derivative is the slope along one axis; the gradient is the assembled uphill arrow; the negative gradient suggests a downhill move whose length you must choose. Optimization is hard not because the definition is subtle but because the terrain is vast, curved, noisy, and observed only through batches.