MQA, GQA & MLA

01 · Concept

Concept

Open Qwen3.8-27B’s config and an asymmetry jumps out: 24 query heads, but only 4 key/value heads. Why would a design team give a layer six times more questions than answers? Because during serving, queries are computed fresh each step and cost only weights — but every key and value must be stored for every past token. KV heads are the expensive ones, and this lesson is about the family of designs that economize on them: MQA, GQA, and MLA.

Set up the accounting. With $H_q$ query heads and $H_{kv}$ KV heads, cache size per token per layer is proportional to $H_{kv}$ (times head dimension, times two for K and V, times bytes per element). Classic multi-head attention sets $H_{kv} = H_q$: every query head owns a private key/value view.

Here is the wrong turn to stage deliberately, because you will see it in the wild. Someone reads “24 heads, head dimension 256, bf16” and computes Qwen’s cache as $24 \times 256 \times 2 \times 2\ \text{B} = 24\ \text{KiB}$ per token per layer. That is the MHA number — it ignores grouped-query attention, and it overestimates by exactly the factor the designers engineered away. The real figure, derived once in lesson 7.2 and only referenced here, uses the 4 KV heads: 4 KiB per token per caching layer, 64 KiB per token across the 16 full-attention layers, 16 GiB per sequence at the native 262,144-token context. Run the hypothetical MHA variant through the same pipeline and the comparison is stark:

a 6× shrink — per token that is 384 KiB versus 64 KiB, and at full context a hypothetical 96 GiB per sequence versus the actual 16 GiB. The MHA variant could not hold even a single full-length sequence next to the 54 GB of bf16 weights on an 80 GB accelerator; the GQA design can. The same factor governs decode bandwidth, since the cache must be read each step by every caching layer.

Multi-query attention (MQA) pushes sharing to the limit: $H_{kv} = 1$. For Qwen’s geometry that would mean $1 \times 256 \times 2 \times 2\ \text{B} = 1\ \text{KiB}$ per token per layer — 16 KiB per token, about 4 GiB at full context, a 24× shrink from MHA. The price is representational: all 24 query heads must read from one shared key/value subspace. Query heads can still ask different questions, but they all consult the same index and the same payload, and whether quality survives depends on model, scale, and training recipe — it cannot be read off the architecture diagram.

Grouped-query attention (GQA) is the interpolation, and Qwen3.8-27B’s choice sits squarely on it: 24 query heads in 4 groups of 6, each group sharing one K/V head. Groups preserve some diversity of key/value subspaces while dividing cache and bandwidth by six. The GQA paper (Ainslie et al.) showed the design can also be converted into: pool an MHA checkpoint’s K/V heads group-wise, then continue training briefly. The mapping is structural — kernels must know which query heads read which K/V head, and checkpoints encode the projection shapes (recall from lesson 4.2 that Qwen’s K and V projections output 1024 dimensions, 4 heads × 256, against 12288 for Q — 6144 of queries interleaved per head with 6144 of gate).

Multi-head latent attention (MLA), introduced with DeepSeek-V2, leaves the $H_{kv}$ axis entirely. Instead of caching per-head keys and values, it caches a learned low-dimensional latent vector per token and reconstructs attention components through projections, with decoupled handling of the RoPE positional part. Its cache size is set by the latent dimension, not a head count, and its algebra differs — the quality-versus-size point it reaches is not on the MHA-GQA-MQA line at all.

Deployment is the final filter. A smaller cache pays off only when kernels, quantization, and parallel layouts implement the exact variant efficiently; falling back to generic attention paths can erase the gain. Compare candidates on cache bytes per token, decode bandwidth, throughput at realistic concurrency, and quality on long contexts — a small-cache design often wins at the system level by admitting more concurrent sequences, a gain single-request benchmarks never see.

The spectrum, with our specimen placed on it: MHA gives every query head a private K/V view (24 KiB per token per layer, in Qwen’s geometry). GQA shares within groups — Qwen3.8-27B’s 24-over-4 buys a 6× shrink and is one half of its memory story, alongside the hybrid layout that limits caching to 16 layers at all. MQA shares one view globally (1 KiB, 24×). MLA re-encodes the problem into a latent cache. Each trades repeated state for a stronger structural constraint — and Qwen’s designers, notably, took two of these savings at once.