Multi-head attention

01 · Concept

Concept

A single attention operation produces one score distribution per query and one mixture of values. Now give it a realistic job. In “The lawyer who signed the contract that the intern drafted was fired,” the position for “was” simultaneously needs its subject (“lawyer,” eight tokens back), agreement information (singular), and the boundary of the intervening clause. One set of projections must compress all of those different notions of “relevant” into a single similarity space — and a single softmax must express all of them in one weighting. Something has to give.

The naive first fix is to make the one head bigger: widen the projections so the single similarity space has room for everything. But a wider dot product still yields one score per key and one distribution per query — a position cannot simultaneously attend sharply to its subject and diffusely to clause boundaries with a single set of weights. The correction is not a wider head but more heads: split the width into independent subspaces, each with its own projections, scores, and softmax. That is multi-head attention.

Here is the worked layout for Qwen3.8-27B’s Gated Attention layers, continuing the shapes from lesson 4.2. The residual stream is 5120-dimensional. The layer forms $h = 24$ query heads of dimension $d_{head} = 256$:

query features per token, reshaped into a head axis from the query half of the gated $5120 \to 12288$ projection (lesson 4.2) — $B \times T \times 6144$ becoming $B \times h \times T \times 256$. Each head $j$ then runs the machinery of lessons 4.3 and 4.4 independently in its own 256-dimensional coordinate system:

with $M$ the causal mask applied identically inside every head. The 24 outputs are concatenated back to width 6144, and the output projection $W_O: 6144 \to 5120$ mixes information across heads before the result rejoins the residual stream. That final mixing matters: without it, head 7’s discoveries would be trapped in head 7’s slice of the concatenation.

Why does this beat one big head at equal parameter count? Because “relevant” no longer has to mean one thing. One head’s subspace can emphasize positional proximity, another’s can match syntactic features, another’s can track an open quotation. Each gets its own softmax, so one head can be nearly one-hot on the subject while another spreads weight across a clause. Later layers compose these routes into behavior no single head could express.

Notice, though, what the paragraph above did not say: that head 3 is “the syntax head.” Michel, Levy, and Neubig showed that in trained models many heads can be pruned with limited immediate damage while a smaller subset is critical; redundancy helps optimization and robustness, and a single function can be smeared across heads and layers. Twenty-four heads are twenty-four opportunities for specialization, not twenty-four labeled experts.

Now the asymmetry that makes this model’s layout genuinely modern. Nothing requires the number of key/value heads to equal the number of query heads. Qwen3.8-27B has 24 query heads but only 4 KV heads — K and V projections of width $4 \times 256 = 1024$, as lesson 4.2 computed. The 24 query heads are partitioned into 4 groups of $24 \div 4 = 6$, and the 6 heads in a group all read the same key and value tensors while keeping their own queries. This is grouped-query attention (GQA): 24 independent ways of asking questions, but only 4 stored sets of answers. The motive is decode-time memory — fewer KV heads means proportionally less cached state per token — and the full cost–benefit analysis, including MQA at one extreme and MLA beyond it, is the business of lesson 7.6, with the cache arithmetic itself in lesson 7.2.

One parameter-count subtlety survives from the classic design: at fixed total width, splitting into more or fewer query heads barely changes parameter count, since the big projections keep the same overall shape — head count changes how the width is partitioned, along with kernel behavior and score-matrix bookkeeping. What GQA changes is different: it shrinks K and V themselves, which is a real reduction, paid for by forcing head groups to share addressing targets.

Multi-head attention is also the launch point for the other half of Qwen’s architecture. The idea that a layer should mix the sequence through many parallel, learned, gated channels does not require softmax attention at all — Qwen’s 48 Gated DeltaNet layers run 48 value heads of their own over a recurrent state instead of a score matrix. That mechanism, and why the model keeps only 16 layers of the full attention described here, is lesson 4.15.

The enduring picture: a bank of learned communication channels, each deciding where to read in its own coordinate system, feeding a shared mixer. The architecture offers specialization, redundancy, and composition — and in its 2026 form, it economizes on the stored half of the conversation.