Self-attention from first principles: Q, K, V

1

Concept

The embedding of a token starts as a context-free lookup. The token “bank” receives the same initial vector in “river bank” and “bank account.” A useful language model needs a representation that changes after reading the surrounding sequence. Self-attention is the Transformer operation that performs this contextual mixing: each position decides how much information to gather from other positions.

Start with a sequence represented as a matrix $X$. Each row is one token position, and each row contains that position’s current features. The attention layer learns three projection matrices, $W_Q$, $W_K$, and $W_V$. Multiplying $X$ by them creates three new matrices:

The names describe roles, not three different input sequences. Queries express what each receiving position is looking for. Keys describe how each available position should be matched. Values contain the information that can be transferred. Every position creates all three from its current representation.

For one receiving position, take its query and compute a dot product with every key. A larger dot product means the vectors point in more compatible directions. Collecting every pair at once gives the score matrix $QK^\top$. The row for one query contains its compatibility with every key. The Transformer scales those scores by the square root of the key dimension and normalizes each row with softmax:

The softmax values are attention weights. They are non-negative and each row sums to one. Multiplying those weights by $V$ produces, for every query position, a weighted sum of value vectors. This is the crucial separation: keys decide where to read, while values determine what is read. If keys and values had to be identical, addressing and payload would be forced into one representation.

Consider “The chef added salt because the soup tasted bland.” The query at “bland” may match the keys for “soup” and “tasted” more strongly than the key for “chef.” Its new representation then mixes value information from those positions. In another sentence, the same word starts from the same embedding but receives different context. Self-attention turns a static token identity into a contextual state.

The word self means queries, keys, and values come from the same sequence. In cross-attention, queries come from one sequence while keys and values come from another, such as a decoder reading encoder states. The same addressing mechanism serves both cases.

Decoder-only language models add a causal mask. When predicting the token at position $i$, that position must not read future tokens. Before softmax, scores for forbidden positions are replaced by a very negative value, which makes their normalized weight effectively zero. Without the mask during training, the model could peek at the answer it is supposed to predict.

One attention calculation is not asked to capture every relationship. Transformers use multiple heads: separate learned projections perform attention in parallel, and their outputs are concatenated and projected again. Different heads can specialize in different useful patterns, although a clean human interpretation is not guaranteed. The full block also includes residual connections, normalization, and a feed-forward network. Attention is a routing operation inside that larger system, not the entire Transformer.

Self-attention has two notable computational properties. First, all pairwise scores can be computed with matrix operations, so training processes positions in parallel instead of stepping through a recurrent state one token at a time. Second, the score matrix has a pair for every pair of positions, giving conventional attention quadratic growth in sequence length. Efficient kernels such as FlashAttention reduce memory traffic without changing the exact mathematical result, while sparse and linear variants alter the computation more substantially.

Attention weights are useful to inspect, but they are not a complete explanation of model reasoning. A high weight shows that one head transferred value information along an edge at one layer. Residual streams, other heads, feed-forward blocks, and later layers can amplify, transform, or cancel that contribution. “The model attended to this token” is evidence about routing, not a proof of causal importance by itself.

The durable mental model is a differentiable lookup system. Every position writes a query, exposes a key, and offers a value. Similarity between queries and keys produces addresses; softmax turns addresses into mixing weights; the weighted values become context. Because the projections are learned end to end, the model discovers what questions, labels, and payloads are useful for predicting language.