Self-attention from first principles: Q, K, V

01 · Concept

Concept

The embedding of a token is a context-free lookup. The token “bank” receives the same initial vector in “river bank” and “bank account,” yet a useful model must treat those two occurrences differently. That is the concrete problem: turn a static token identity into a representation that depends on the surrounding sequence. Self-attention is the operation that does it — each position decides how much information to gather from every other allowed position. The positions themselves are tokenizer outputs from lesson 1.2; in Qwen3.8-27B each one is a vector of 5120 numbers, the model’s hidden size.

Represent the sequence as a matrix $X$: one row per token position, 5120 features per row. The attention layer learns three projection matrices, $W_Q$, $W_K$, and $W_V$, producing

The names describe roles, not different inputs. Queries express what each receiving position is looking for. Keys describe how each available position should be matched. Values contain the information that can be transferred. Every position creates all three from its current representation.

Now the worked example, with real shapes. A reasonable first guess — and the classic wrong one — is that each projection maps the hidden size to itself: $5120 \to 5120$, a square matrix. Qwen3.8-27B’s configuration refutes it. The model’s Gated Attention layers use 24 query heads, each of dimension 256. The query projection must produce all of them, so the query state is head count times head dimension:

That 6144 is the width of the query state, and it is genuinely non-square — $6144 - 5120 = 1024$ wider than a square matrix would be. The weight matrix that produces it is wider still. These are gated attention layers (attn_output_gate in the config), and the gate is fused into the same projection: q_proj emits head count times head dimension twice over, $24 \times 256 \times 2 = 12288$. The split is per head, not down the middle — the output is viewed as 24 heads of $256 \times 2$, and each head’s last axis is cut into 256 query values and 256 gate values. Gather the query halves and you have the 6144-wide query state; gather the gate halves and you have a second 6144 that never enters the score matrix. Instead, once attention has run, its output is scaled elementwise by $\sigma(\text{gate})$ and only then projected. So on disk the tensor is $5120 \to 12288$; only half of it ever becomes a query. Keys and values are narrower still: the model has only 4 KV heads (a grouped-query design covered in lesson 7.6), so

and both the key and value projections are $5120 \to 1024$. After the heads compute their outputs and are concatenated back to width 6144, an output projection maps $6144 \to 5120$ so the result fits the residual stream again. Four rectangles, no squares: $q: 5120 \to 12288$ (queries and gate together), $k: 5120 \to 1024$, $v: 5120 \to 1024$, $o: 6144 \to 5120$. Note that $o$‘s input is 6144 wide, not 12288: what reaches it is the concatenated, gate-scaled attention output, which happens to share the query state’s width. The gate is applied elementwise just before $o$, so it scales what $o$ receives without widening it.

With shapes settled, the mechanism: for one receiving position, take its query and compute a dot product with every key. A larger dot product means more compatible directions. All pairs at once form the score matrix $QK^\top$. Scale by the square root of the head dimension (the subject of lesson 4.3) and normalize each row with softmax:

The softmax outputs are attention weights: non-negative, each row summing to one. Multiplying by $V$ gives every query position a weighted sum of value vectors. This is the crucial separation — keys decide where to read, values determine what is read. If keys and values were forced to be the same vectors, addressing and payload would share one representation and each would compromise the other.

Consider “The chef added salt because the soup tasted bland.” The query at “bland” may match the keys for “soup” and “tasted” more strongly than the key for “chef.” Its new representation then mixes value information from those positions. In another sentence the same token starts from the same embedding but ends somewhere different. That is the contextual mixing we set out to build.

The word self means queries, keys, and values come from one sequence. Decoder-only models like Qwen3.8-27B add a causal mask — position $i$ must not read positions after $i$ — which lesson 4.4 develops. And one attention calculation is not asked to capture every relationship: the projections are split into the 24 parallel heads mentioned above, the subject of lesson 4.5. The full block adds residual connections, normalization, and a feed-forward network; attention is the routing operation inside that larger system.

Two computational properties matter later. First, all pairwise scores are matrix operations, so training processes positions in parallel. Second, the score matrix has an entry for every pair of positions, giving quadratic growth in sequence length — the bill lesson 4.3 quantifies at this model’s real context length and track 7 works to reduce.

Attention weights are evidence about routing, not proof of reasoning. A high weight shows one head transferred value information along one edge at one layer; residual streams, other heads, and later layers can amplify or cancel it. The durable mental model is a differentiable lookup: every position writes a question, wears a label, and offers a payload, and the projections that define all three are learned end to end.