MQA, GQA & MLA

1

Concept

In ordinary multi-head attention (MHA), each head has its own query, key, and value projections. During autoregressive serving, every past position therefore contributes a key and value for every head. The cache and bandwidth can become a major cost. MQA, GQA, and MLA change how K/V information is represented without removing the multiplicity of queries.

Let there be $H_q$ query heads and $H_{kv}$ key/value heads. Ignoring other dimensions, KV-cache size is proportional to $H_{kv}$. MHA uses $H_{kv}=H_q$. Multi-Query Attention (MQA) uses $H_{kv}=1$: all query heads attend to the same keys and values. Query heads can still learn different questions, but they read from a shared address and payload representation.

This sharing dramatically reduces cached tensors and the bytes read during decode. The trade-off is capacity. Independent K/V heads can represent different subspaces; forcing all queries through one K/V set may degrade quality for a given model and training recipe. The effect cannot be inferred from architecture alone and must be evaluated.

Grouped-Query Attention (GQA) chooses an intermediate $1<H_{kv}<H_q$. Query heads are divided into groups, and each group shares one K/V head. If several query heads express related reading patterns, sharing preserves much of the benefit while reducing cache. GQA can be trained from scratch, and the 2023 GQA paper also studied converting multi-head checkpoints by pooling K/V heads followed by additional training.

The group mapping is structural. Kernels need to know which query heads use each K/V head; checkpoints encode projection shapes accordingly. Changing a configuration field without converting weights is not an optimization—it is a broken model. Tensor-parallel layouts also need compatible divisibility or explicit sharding rules.

Multi-head Latent Attention (MLA), described in DeepSeek-V2 in 2024, takes another route. It compresses key/value information into lower-dimensional latent representations and uses learned projections to recover attention components. Its treatment of positional information, including decoupled components associated with RoPE, is part of the design. MLA should not be summarized as “GQA with more compression”; its cache contents and algebra differ.

The deployment question is end-to-end support. A theoretically smaller cache helps only if kernels, quantization, parallelism, and model loaders implement the exact architecture efficiently. Falling back to generic operations can erase gains. Converting weights into an unsupported variant is not lossless unless a validated conversion and adaptation procedure exists.

Compare variants using cache bytes per token, decode bandwidth, throughput at relevant concurrency, TTFT, output quality, and long-context behavior. Report precision and cache dtype. A small-cache architecture can enable larger batches, producing a system-level gain that a single-request benchmark misses.

The durable spectrum is straightforward. MHA preserves a separate K/V view for every query head. MQA shares one view globally. GQA shares within groups. MLA stores a compressed latent view and reconstructs what attention needs according to its architecture. Each reduces repeated state in exchange for a stronger representational constraint and more specialized implementation.