Multilayer perceptrons and non-linearity

1

Concept

A multilayer perceptron, or MLP, composes learned affine transformations with nonlinear activation functions. For a batch matrix $X$, a one-hidden-layer network may compute

Rows are examples, columns are features, and $\phi$ acts element by element. Training chooses weights and biases so the final output supports a task.

The activation is not decoration. Without it, $W_2(W_1x+b_1)+b_2$ can be rearranged into one affine transformation $Wx+b$. Ten such layers still define one hyperplane for binary classification. A nonlinear activation breaks this algebraic collapse. ReLU, for example, keeps positive coordinates and sets negative ones to zero, creating different linear behavior in different regions of input space.

This piecewise structure solves XOR. One hidden unit can activate for one relevant half-space and another for a different one. The output combines their signals to isolate diagonally arranged positives. The exact construction is less important than the principle: hidden layers learn a new coordinate system in which the classes may become linearly separable. Representation learning replaces hand-designed transformed features.

Width is the number of units in a hidden layer; depth is the number of successive transformations. More width allows more features at a stage. Depth can reuse intermediate features compositionally, representing some functions more efficiently than a single enormous layer. Neither guarantees learning. Optimization may fail, data may not identify the intended function, and excessive capacity may memorize noise.

Universal approximation results are frequently oversold. Under stated conditions, a sufficiently wide network with a suitable nonlinearity can approximate continuous functions on a compact domain arbitrarily well. The theorem does not say how many units are needed, that gradient descent will find them, that finite data selects the right function, or that the result generalizes outside the domain. Expressibility is only one part of learning.

MLPs are trained by defining a loss, computing gradients through each operation, and updating parameters. Batches make matrix operations efficient and provide noisy estimates of the dataset gradient. Initialization affects signal scale; normalization can stabilize intermediate states; residual connections give gradients shorter paths in deep networks. These engineering choices become essential as depth grows.

For classification, the final layer often emits logits rather than thresholded decisions. Binary cross-entropy or multiclass cross-entropy supplies a differentiable objective. During evaluation, probabilities or logits can be converted into decisions under a policy appropriate to error costs. Confusing the training objective with the deployment threshold leads to poorly calibrated systems.

Transformers contain an MLP inside every block, commonly called the feed-forward network. It acts independently at each token position while attention moves information between positions. Its matrices expand and contract the feature dimension, and its activation provides nonlinearity and gating. The humble MLP is therefore not merely a historical stepping stone: it is half of the repeated computational core of modern language models.

Debugging an MLP begins with a tiny batch it should be able to memorize. Check every tensor shape, confirm the loss falls, and inspect whether activations or gradients are entirely zero or non-finite. That test does not prove generalization, but failure reveals a wiring, objective, or optimization problem before a large run conceals it. Then compare against a linear baseline: if the MLP adds no held-out value, extra depth may be unnecessary.