Vectors, matrices, and dot products

01 · Concept

Concept

Open the configuration file that ships with Qwen3.8-27B and one entry appears more often than any other in this course: the hidden size is 5120. Everything in the next several tracks is a consequence of that number, so it is worth knowing exactly what it means before meeting any terminology. It means that inside the model, at every one of its 64 layers, each token of your prompt is represented by a list of 5120 numbers. Not a word, not a symbol, not a lookup key. Five thousand one hundred and twenty floating-point values. The lesson’s job is to make that object ordinary.

A vector is an ordered list of numbers, and order is part of its identity: $(2,5)$ is not $(5,2)$. In geometry a two- or three-dimensional vector draws as an arrow with direction and length. In machine learning it usually has hundreds or thousands of coordinates and represents a token, an image patch, a hidden state, or a gradient. You cannot picture 5120 axes, and you do not need to. Every operation below is defined coordinate by coordinate and behaves the same at any width.

Vectors add component by component and scale by a number. If $a=(1,2)$ and $b=(3,-1)$, then $a+b=(4,1)$, and $3a=(3,6)$. The Euclidean length is $\lVert a \rVert = \sqrt{1^2+2^2}$. Magnitude and direction carry different information, and neural networks are constantly reshaping both. A Qwen hidden state is exactly this object with 5120 slots instead of two; in bfloat16 it occupies two bytes per number, which is ten kibibytes for a single token at a single layer.

The dot product collapses two equal-length vectors into one scalar:

For the vectors above, $a \cdot b = 1\times3 + 2\times(-1) = 1$. Each matching pair contributes positive evidence when the signs agree and negative evidence when they oppose. Geometrically, $a\cdot b = \lVert a\rVert \lVert b\rVert \cos\theta$: aligned vectors score positively, perpendicular ones score zero, opposed ones score negatively. This single operation is the atom of everything downstream. Attention scores are dot products between queries and keys. A logit is a dot product between a hidden state and one row of the output matrix. When lesson 0.1 said the model produces 248,320 scores, it meant it performs 248,320 dot products, each against a 5120-dimensional state.

Now a wrong turn worth taking on purpose. Suppose you are comparing two token representations and one pair scores 340 while another scores 12. It is tempting to conclude the first pair is far more related. Work it out instead. Let $u = (100, 0)$ and $v = (3.4, 3.4)$: their dot product is $340$, though the angle between them is 45 degrees. Let $s = (1,1)$ and $t = (6,6)$: their dot product is $12$, and the angle is exactly zero, perfect alignment. The large score came from length, not agreement. Cosine similarity repairs this by dividing the lengths out, $(a\cdot b)/(\lVert a\rVert\lVert b\rVert)$, giving $0.707$ for the first pair and $1.0$ for the second. Embedding comparisons in lesson 1.5 usually want cosine; attention deliberately keeps the raw scaled dot product, because there magnitude is part of the learned computation rather than a nuisance.

A matrix is a rectangular array of numbers, written rows by columns. A $3 \times 4$ matrix maps a four-component input to a three-component output, and each output coordinate is the dot product of one matrix row with the input. That is why matrix-vector multiplication is the workhorse: it computes many learned weighted questions about the same input at once. Inside our specimen, the projection that produces attention queries takes the 5120-dimensional state and emits a 12288-dimensional one — 24 heads of 256 dimensions each, and then the same again for a gate that rides beside them. That mismatch is a real property of the model and not a typo, and lesson 4.2 explains why the temptation to round it back to 5120 wrecks the arithmetic.

Shapes function as a type system. An $m\times n$ matrix times an $n$-vector yields an $m$-vector, and the inner dimensions must agree. For batched data, rows usually index examples or token positions, so a matrix of shape sequence-by-features multiplies a weight matrix of shape features-by-output. Writing the shapes beside the equation catches most bugs before the code runs, and it is the single habit that most reliably separates people who can read model code from people who cannot.

The bridge to the rest of the course is now short. An embedding lookup returns a 5120-dimensional vector for a token identifier. Projection matrices turn that state into queries, keys, values, or vocabulary logits. Query-key dot products create attention scores. Gradients are vectors pointing toward local change in the loss. Once vectors are states, matrices are learned maps, and dot products are weighted alignment, most of a Transformer reads as linear algebra with a few carefully placed nonlinear interruptions.