Backpropagation, visualized

01 · Concept

Concept

Lesson 2.4 produced a single number: the loss. To improve the model you need to know, for every parameter, whether nudging it up or down would make that number smaller. There is an obvious way to find out. Perturb one parameter by a tiny amount, run the model again, see how the loss moved, and divide. Repeat for the next parameter.

Price that plan honestly, because it is the wrong turn worth walking into. A careful two-sided estimate needs two forward passes per parameter. For a small network with ten thousand parameters that is twenty thousand forward passes per gradient — slow but survivable. For Qwen3.8-27B, with roughly 27 billion parameters (Qwen3.8-27B Model Card, 2026), it is 54 billion forward passes to obtain the gradient for a single training step. If one forward pass took a millisecond, one step would take about twenty months. Finite differences do not scale badly; they do not scale at all.

Backpropagation is what replaces it, and the improvement is not incremental. One backward sweep produces the gradient for every parameter at once, at a cost of roughly the same order as evaluating the graph. A larger model still makes both passes more expensive; the saving is that reverse mode does not require a separate evaluation for every parameter.

The mechanism is the chain rule applied to a recorded graph. The forward pass evaluates operations in order, saving the intermediate values that derivatives will need. The backward pass starts at the end with $\partial L/\partial L=1$ and walks the operations in reverse, and at every operation the rule is the same: upstream gradient times local derivative. Each operation only needs to know how its own outputs respond to its own inputs; it never needs to know anything about the rest of the network.

Do one by hand, using the loss from lesson 2.4. Take three logits $z=(2.0,\,1.0,\,0.1)$ with the observed class being the first. Forward: exponentials are about $7.389$, $2.718$, and $1.105$, summing to $11.212$, so $p\approx(0.659,\,0.242,\,0.099)$ and $L=-\ln 0.659\approx0.417$. Backward: for softmax followed by cross-entropy the gradient with respect to the logits collapses to the beautifully simple $\partial L/\partial z=p-y$, which here is $(0.659-1,\;0.242-0,\;0.099-0)=(-0.341,\,0.242,\,0.099)$. Read it as an instruction. The correct class carries a negative gradient, so gradient descent will raise its logit. The two wrong classes carry positive gradients in exact proportion to the probability they took, so their logits get pushed down, the more confidently wrong one hardest. Now suppose those logits came from $z=Wh+b$ for a hidden vector $h$. The same rule continues: $\partial L/\partial b=p-y$, $\partial L/\partial W=(p-y)h^{\top}$, and $\partial L/\partial h=W^{\top}(p-y)$ — that last one is the message handed to the layer below, which will apply the identical procedure with its own local derivative.

Graphs branch, and branching is where implementations get subtle. If a value influences the loss along two paths, its gradient is the sum of the two contributions, because that is what the multivariable chain rule says. This is why autodiff frameworks accumulate into gradient buffers rather than overwriting them, and consequently why a training loop must zero those buffers at batch boundaries: summation is the mathematically correct default for repeated use, and only the loop knows where one batch ends.

The bill arrives as memory rather than time. The backward pass needs values the forward pass computed, so training holds activations alive from the moment they are produced until their gradients are consumed. Multiply that by 64 layers, by the intermediate width of 17408 inside each feed-forward block, by every token position in the batch, and activation memory rivals or exceeds parameter memory. Gradient checkpointing buys memory back by storing only selected boundaries and recomputing the rest during backward — more compute for less memory, and a lever lesson 5.10 turns explicitly.

Two habits keep this honest. For any new custom operation, gradient-check it against finite differences on a tiny input — the method that fails at scale is perfectly good as a unit test, given sensible precision and step size, away from non-differentiable points. And before a long run, confirm that every parameter you expect to train has a finite, nonzero gradient after one loss. The enduring picture is a reverse flow of responsibility through a recorded graph, where purely local rules compose into global sensitivity, and where the optimizer of lesson 2.7 has not yet done anything at all.