Matrix multiplication as transformation

01 · Concept

Concept

You have a token’s hidden state: 5120 numbers, as lesson 0.4 established. Inside every layer of Qwen3.8-27B, that state gets pushed through a feed-forward block that first widens it to 17408 dimensions and then narrows it back. Both of those numbers come straight out of the model’s configuration. The question this lesson answers is what that widening actually does, and what it costs, because the answer explains most of where a 27-billion-parameter model’s parameters and arithmetic actually go.

Matrix multiplication is easiest to understand as a transformation rather than as a grid-filling ritual. A matrix $W$ takes coordinates in one space and produces coordinates in another. Each output component of $Wx$ is a dot product between one row of $W$ and the input $x$. The rows are therefore independent learned questions asked about the same input, and the number of rows is how many questions get asked.

A small case makes the geometry visible. Take

Applied to $(x_1,x_2)$ it yields $(2x_1, x_2/2)$: a square becomes a rectangle, stretched horizontally and squashed vertically. Other matrices rotate, reflect, shear, or project onto a subspace. At 5120 dimensions the picture disappears, but the operation is unchanged — a learned change of representation, nothing more exotic.

Now the real one. With the convention used here, $Wx$, Qwen3.8-27B’s up-projection has shape $17408 \times 5120$: one row for each output and one column for each input. To transform a single token, the machine must produce 17408 output coordinates, and each one is a dot product over 5120 inputs. Count it directly:

That is one matrix, one layer, one token. The model has 64 layers, and the feed-forward block of each layer contains more than one matrix of comparable size; lesson 4.9 does the full accounting and shows that the feed-forward blocks hold the large majority of the model’s parameters. But hold the single number first, because it recalibrates intuition. Generating a hundred-token reply is not a hundred small lookups. Every token pushed through every layer triggers tens of millions of multiply-adds in this matrix alone, and the hardware questions of track 9 exist because of it.

The rank of a matrix measures how many independent directions actually pass through it. A low-rank map forces its outputs into a smaller subspace than the shape suggests. This is not merely a caveat: it is the mechanism behind parameter-efficient fine-tuning in lesson 6.3, which adapts a huge matrix by adding a deliberately low-rank correction rather than retraining all 89 million numbers.

Matrix-matrix multiplication does many of these transformations at once. Entry $(i,j)$ of $AB$ is the dot product of row $i$ of $A$ with column $j$ of $B$. More important than the recipe is the meaning: $AB$ represents composition, with $B$ acting first and $A$ second. Multiplication is not commutative. $AB$ and $BA$ can differ, and one can be defined while the other is not. Shape is what records which compositions make sense.

Neural layers usually compute $y = Wx + b$. The matrix supplies the linear transformation and the bias translates every result, making the map affine rather than strictly linear: zero can now map to $b$ instead of to zero. That freedom lets a decision boundary sit somewhere other than through the origin.

Here is the classic wrong turn, and it is the reason activation functions exist at all. Suppose you stack two linear layers to get depth: $W_2(W_1 x)$. By associativity that equals $(W_2 W_1)x$, and $W_2 W_1$ is just another matrix. Two layers have exactly the expressive power of one. Add biases and the same collapse happens with a single combined affine map. Stack fifty of them and you still have one matrix. Depth in a purely linear network is an illusion, purchased with parameters and paid for in arithmetic, delivering nothing. Something nonlinear must interrupt the composition, which in Qwen’s feed-forward block is a gated activation covered in lesson 2.3.

Matrix multiplication dominates neural computation partly because it is expressive and largely because hardware is exceptionally good at it. A batch of token vectors multiplied by one weight matrix reuses the same parameters across every position, so the arithmetic-to-memory ratio is favorable. Accelerators split large products into tiles, stream them through a memory hierarchy, and accumulate partial sums. The algebra is simple; whether it runs fast depends on shapes, precision, data movement, and fusion, which is the subject of tracks 8 and 9.

The reading habit to take forward is a question asked at every matrix in a model: which space enters, which space leaves, how many independent directions can survive, and what does this cost per token? Applied to Qwen’s projection from 5120 inputs to 17408 outputs, the answers are a narrow state in, a wide state out, at most 5120 independent directions, and 89.1 million multiply-adds. That is what a transformer layer feels like from the inside.