The perceptron

01 · Concept

Concept

Start with a problem that has nothing to do with neural networks. A bag arrives at a museum entrance and you have two measurements: the number of metal objects detected, and whether the visitor carries a verified equipment tag. You must decide, from those two numbers alone, whether to inspect the bag. You could write rules by hand. Instead, suppose you are given a few hundred past bags with the correct decision attached, and you want the machine to find the rule. What is the simplest device that can learn one?

The answer, from 1958, is the perceptron. It takes a feature vector $x$, stores one number per feature in a weight vector $w$, adds a bias $b$, and computes a single score

If $z$ is positive it predicts one class, if negative the other. That is the whole device: a weighted sum and a threshold.

Each weight says how one feature moves the score. A positive weight raises the score as its feature grows; a negative weight lowers it. Magnitude reflects sensitivity in whatever units the feature arrives in, which is why feature scaling matters — a measurement that ranges over thousands will dominate one that ranges over zero to one, whether or not it is more informative. The bias is the intercept, the score when every feature is zero. It slides the whole policy toward leniency or strictness.

Work one example through by hand. Let metal count be $x_1$ and equipment tag be $x_2\in\{0,1\}$, and suppose training has landed on $w=(0.6,-1.5)$ with $b=-1.2$. A bag with four metal objects and no tag scores $0.6\times4-1.5\times0-1.2=1.2$, which is positive, so it is inspected. The same four objects carried by a tagged technician score $0.6\times4-1.5\times1-1.2=-0.3$, which is negative, so it passes. The tag weight is large enough to cancel two and a half metal objects. In two dimensions the boundary $0.6x_1-1.5x_2-1.2=0$ is a straight line, the vector $w$ points perpendicular to it, changing the weight ratio rotates it, and changing $b$ slides it without rotation. In more dimensions the same equation describes a hyperplane.

Learning is mistake-driven. For a misclassified example $(x,y)$ with labels encoded as $-1$ and $+1$, one classic form updates $w\leftarrow w+\eta y x$ and $b\leftarrow b+\eta y$, with learning rate $\eta>0$. A positive example wrongly predicted negative pulls the weight vector toward that input; a negative example wrongly predicted positive pushes it away. Correctly classified examples trigger nothing. If the data really is linearly separable, the perceptron convergence theorem guarantees this procedure stops making mistakes after finitely many updates. It does not promise the widest margin, calibrated probabilities, or good behavior on new bags.

Now the classic wrong turn, and it is worth walking into deliberately. Take the XOR pattern: label $(0,0)$ and $(1,1)$ as one class, $(0,1)$ and $(1,0)$ as the other. Train the perceptron; it oscillates. The instinctive diagnosis is an optimization problem — lower the learning rate, run more epochs, try a different initialization. All of that is wasted effort. Draw the four points and try to separate the diagonals with a ruler: it cannot be done. The failure is representational, not numerical. No values of $w$ and $b$ exist that classify all four correctly, so no procedure for finding values can succeed. Fixing it requires changing the representation — adding a transformed feature such as $x_1x_2$, or composing several units with a nonlinearity between them, which is exactly the move lesson 2.2 makes.

The hard threshold has a second defect that shaped everything after it. Its derivative is zero almost everywhere and undefined at the boundary, so gradient-based learning has nothing to work with. Modern classifiers keep the linear score and replace the step with something differentiable — a sigmoid, or the softmax of lesson 0.3 — then apply a decision threshold afterward if a decision is needed at all.

So the perceptron is not merely a museum piece. It teaches the geometry that survives at every scale: learned weights define a direction, the bias defines an offset, a dot product scores compatibility, and a model can only separate what its representation makes separable. When you inspect any fitted linear boundary, plot the examples and the margin rather than reporting accuracy alone — a separator can classify a small sample perfectly while sitting so close to every point that measurement noise flips the decision.