The perceptron

1

Concept

The perceptron is a small model with a large historical footprint. It receives a feature vector $x$, stores one weight per feature in $w$, adds a bias $b$, and computes $z=w\cdot x+b$. A threshold turns that score into one of two classes. With labels encoded as $-1$ and $+1$, the prediction is the sign of $z$.

Each weight controls how one feature contributes. A positive weight raises the score as its feature increases; a negative weight lowers it. The absolute magnitude reflects sensitivity in the units presented to the model. The bias is an intercept: it changes the baseline score when all input features are zero. Feature scaling matters because a large numerical range can dominate even when its underlying property is not more informative.

In two dimensions, the equation $w_1x_1+w_2x_2+b=0$ describes a line. Points on one side produce positive scores and points on the other produce negative scores. The vector $w$ is perpendicular to that line. Changing the weight ratio rotates the boundary; changing $b$ shifts it without rotation. In more dimensions, the same boundary is a hyperplane.

The classic perceptron learning rule reacts to mistakes. For a misclassified example $(x,y)$, one form updates $w\leftarrow w+\eta yx$ and $b\leftarrow b+\eta y$, with learning rate $\eta>0$. A positive example predicted negative pulls the weight vector toward that input; a negative example predicted positive pushes it away. Correct examples need no update under the basic rule.

If a dataset is linearly separable, the perceptron convergence theorem says this mistake-driven procedure reaches some separating hyperplane after finitely many mistakes under standard assumptions. It does not promise the widest margin, calibrated probabilities, or good generalization. The chosen separator depends on data order, scaling, initialization, and updates. Separability is a property of the supplied representation, not necessarily of the underlying world.

XOR exposes the limitation. Label $(0,0)$ and $(1,1)$ one class, and $(0,1)$ and $(1,0)$ the other. No line divides the diagonal pairs correctly. Training longer cannot make one perceptron represent a non-linear boundary. Adding useful transformed features or composing multiple units with a nonlinearity changes the representation and can solve it. That step leads directly to multilayer networks.

The hard threshold also has zero derivative almost everywhere and is discontinuous at the boundary, which makes ordinary gradient-based learning unsuitable. Modern binary classifiers often keep the linear score but use a differentiable sigmoid during training and a probabilistic loss. The decision may still be thresholded later. Separating training score, probability interpretation, and final policy threshold avoids confusion.

Language models contain vastly more machinery, yet perceptron logic remains visible. A neuron computes a weighted sum plus bias before an activation. Linear classification heads map hidden vectors to logits. Dot products score compatibility. The perceptron teaches the essential geometry: learned weights define a direction, bias defines an offset, and a model can only separate patterns made accessible by its representation.

When inspecting a fitted boundary, plot the examples and margins rather than reporting accuracy alone. A separator can classify the small training sample perfectly while lying dangerously close to every point, making tiny measurement noise reverse decisions.