Backpropagation, visualized

1

Concept

Backpropagation is an efficient application of the chain rule to a computational graph. It answers: for the scalar loss produced by this forward computation, how sensitive is the loss to every intermediate and parameter? It does not choose the loss, collect data, or update weights. Those are separate responsibilities.

Consider $a=wx+b$, $h=\operatorname{ReLU}(a)$, and $L=(h-y)^2$. The forward pass evaluates these operations in order and saves values needed for derivatives. The graph has variables as nodes and operations as edges or operation nodes. For one input, it produces a numerical loss and a trace of how that loss was constructed.

The backward pass begins with $\partial L/\partial L=1$. For the square, the local derivative is $2(h-y)$, producing $\partial L/\partial h$. ReLU passes that gradient when $a>0$ and blocks it when $a<0$. The affine operation then yields $\partial L/\partial w=(\partial L/\partial a)x$, $\partial L/\partial x=(\partial L/\partial a)w$, and $\partial L/\partial b=\partial L/\partial a$.

The pattern is upstream gradient times local derivative. Each operation knows how its outputs change with its inputs. Reverse-mode automatic differentiation visits operations in reverse topological order, multiplying or applying vector-Jacobian products. It never needs to materialize an enormous full Jacobian for a scalar loss. This makes reverse mode especially efficient when there are many parameters and one objective.

Graphs can branch. If a value $x$ influences loss along two paths, its derivative is the sum of both contributions. This is why autodiff frameworks accumulate gradients. It is also why PyTorch parameter gradients accumulate across repeated backward calls unless cleared: summation is the mathematically correct default for multiple uses, while the training loop decides where batch boundaries lie.

Memory is a central trade-off. Backward often needs forward activations, so training stores them until their gradients are computed. Gradient checkpointing saves selected states and recomputes missing intermediates during backward, exchanging extra compute for lower activation memory. In-place mutation can destroy values the graph expects; frameworks detect some cases, but disciplined code avoids surprising aliasing.

Gradient checking validates a small implementation with finite differences: perturb one parameter by $+\epsilon$ and $-\epsilon$, estimate the slope, and compare it with backprop. The check requires suitable precision and step size and can fail near nondifferentiable points. It is too expensive for training but excellent for testing a new custom operation.

In deep networks, products of many local derivatives can shrink or grow, producing vanishing or exploding gradients. Activation choice, initialization, normalization, residual connections, clipping, and architecture all influence the path. Backprop faithfully reports the derivative of the computation it is given; a useless gradient can reflect poor geometry rather than a bug in differentiation. The enduring picture is a reverse flow of responsibility through a recorded graph, with local rules combining into global sensitivity.

Autodiff also follows the graph actually executed, including branches and detached values. Accidentally converting a tensor to a plain number, mutating saved state, or running a region under no_grad can sever a path. Inspecting whether expected parameters have finite, nonzero gradients is therefore part of validating the program, not an optional optimization ritual.