Multilayer perceptrons and non-linearity

01 · Concept

Concept

Lesson 2.1 ended at a wall: XOR. Four points, two classes, and no straight line that separates them. The device could not be tuned into a solution because it could not express one. So the practical question this lesson answers is narrow and concrete — what is the smallest change to the perceptron that makes XOR representable, and does that change scale to anything larger?

The change is to stack. Feed the input to several perceptron-like units at once, pass each unit’s score through a nonlinear function, and let a second layer of units read those outputs instead of the raw input. For a batch matrix $X$ whose rows are examples and columns are features, a one-hidden-layer network computes

where $\phi$ acts element by element. This is a multilayer perceptron, or MLP.

Everything depends on $\phi$ actually being there. Delete it and the algebra collapses: $(XW_1+b_1)W_2+b_2$ rearranges into $XW+b$ with $W=W_1W_2$ and $b=b_1W_2+b_2$. Ten layers, a hundred layers, any number — still one affine map, still one hyperplane, still exactly as helpless on XOR as the single perceptron was. Depth without nonlinearity is not a weaker version of depth; it is a more expensive way to write the same function.

Watch the fix work on XOR itself, by hand, with no training involved. Use two hidden units with ReLU, which keeps positive values and zeroes negatives: $h_1=\operatorname{ReLU}(x_1+x_2)$ and $h_2=\operatorname{ReLU}(x_1+x_2-1)$. Evaluate the four corners in turn. At $(0,0)$: $h_1=0$ and $h_2=\operatorname{ReLU}(-1)=0$. At $(0,1)$ and at $(1,0)$: $h_1=1$ and $h_2=\operatorname{ReLU}(0)=0$. At $(1,1)$: $h_1=2$ and $h_2=\operatorname{ReLU}(1)=1$. Now let the output unit compute $y=h_1-2h_2-0.5$. The corners give $-0.5$, then $+0.5$ and $+0.5$, then $2-2-0.5=-0.5$. Positive exactly on the two “different” corners, negative exactly on the two “same” corners: XOR, solved.

Look at what happened geometrically. In the new $(h_1,h_2)$ coordinates the four points sit at $(0,0)$, $(1,0)$, $(1,0)$, and $(2,1)$. The two different-input, XOR-positive corners collapse to the same hidden point $(1,0)$, while the two equal-input, XOR-negative corners land at $(0,0)$ and $(2,1)$. One straight boundary now separates the shared positive point from both negative points. The output unit is still a perceptron, unchanged from lesson 2.1. The hidden layer changed what it is looking at.

That is the durable idea: hidden layers learn a representation, and classification happens in the learned space rather than the raw one. Hand-designed feature engineering becomes learned feature construction.

Width is how many units a layer has; depth is how many successive transformations there are. Width buys more features at one stage; depth lets later stages reuse earlier features compositionally, which can represent some functions far more compactly than one enormous layer. Neither guarantees anything on its own.

Now the reason this matters for a course about language models. Every layer of a Transformer contains an MLP, conventionally called the feed-forward network, and it is not a minor component. In Qwen3.8-27B the hidden state carries 5120 numbers per token; the feed-forward block expands that to an intermediate width of 17408, applies a gated activation which lesson 2.3 dissects, and projects back down to 5120 (Qwen3.8-27B Model Card, 2026). One matrix of shape 5120 by 17408 is about 89.1 million multiply-adds for a single token. The full gated block — gate, up, and down projections together — is roughly 267 million parameters per layer, and the model has 64 layers, so the feed-forward blocks alone account for something on the order of 17 billion of the model’s 27 billion parameters.

There is a real division of labor here worth fixing early. The feed-forward block never looks at other tokens. It receives one 5120-dimensional vector, transforms it, and returns a vector of the same size. Whatever it knows about the rest of the sentence arrived through attention before it ran.

Training an MLP follows the same loop everywhere: define a loss (lesson 2.4), compute gradients through every operation (lesson 2.5), update parameters (lesson 2.7). For classification the final layer emits logits, not decisions, and cross-entropy supplies a differentiable objective; converting logits into an action is a separate policy question that belongs to deployment, not training.

Debug an MLP by asking it to memorize a tiny batch it should trivially fit. Check every tensor shape, confirm the loss falls to near zero, and inspect whether any activation or gradient is entirely zero or non-finite. That test proves nothing about generalization, but failing it reveals a wiring, objective, or optimization bug before a long run buries it. Then compare against a plain linear baseline: if the extra depth adds no held-out value, it was not earning its parameters.