Positional encoding III: ALiBi & relative bias

01 · Concept

Concept

There is a third place position can live. Lesson 4.6 put it in the input, added to embeddings. Lesson 4.7 put it in the score’s geometry, rotated into queries and keys. This lesson’s family puts it in the score’s value: compute content compatibility first, then adjust the number directly according to how far apart the two positions are. Historically this road was seriously traveled — T5 and several prominent LMs shipped on it — and understanding it sharpens what the winning road actually won.

The concrete problem, stated as a modeling prior: most useful information for predicting a token sits nearby, but “nearby matters more” is nowhere in the machinery of lessons 4.2–4.5. Could we just tell the model? Let $s_{ij}=q_i\cdot k_j/\sqrt{d_k}$ be the content score from query position $i$ to key position $j$, and add a bias:

Softmax then normalizes the adjusted scores; routing probabilities change while value payloads stay untouched. The function $b$ is the design space. T5 learns it, bucketed by offset: small offsets get individual buckets, large distances share coarse ones, one learned value per bucket per head — a bounded parameter budget encoding the prior that the difference between 1 and 2 tokens matters more than between 1,001 and 1,002.

ALiBi strips even the learning away. In causal attention each head gets a fixed linear penalty into the past:

with slope $m_h$ varying across heads. A worked example shows the texture. Take a head with slope $m_h = 0.5$ scoring three keys at distances 1, 4, and 8 whose content scores are all equal at 2.0. The biased logits are $2.0 - 0.5 \times 1 = 1.5$, then $2.0 - 0.5 \times 4 = 0$, then $2.0 - 0.5 \times 8 = -2.0$: a strong recency preference, though a distant key with much higher content score could still win. A sibling head with slope $m_h = 0.0625$ charges only $-0.0625$, $-0.25$, and $-0.5$ at those same distances — nearly indifferent, keeping long-range routes open. Steep heads become local specialists, shallow heads long-haul channels, and content can always outvote the tariff. Nothing is a hard window; everything is a graded prior.

The classic implementation mistake is where to apply the penalty. The tempting wrong version — attention weights are what we want to shrink, so subtract from the softmax outputs — breaks normalization: rows no longer sum to one, and a subtraction of probabilities is not equivalent to any bias of logits. The correction is the one already in the formula: the bias joins the logits, before softmax, exactly like the causal mask’s $-\infty$ in lesson 4.4 — of which ALiBi is really a soft, sloped cousin.

Why, then, is this largely the road not taken for modern decoder LMs? The bias interface has genuine virtues — clean separation of “how compatible” from “how distant,” visible locality in the attention matrix, near-zero parameter cost. But it can express only what a scalar function of distance can express: a monotone or bucketed preference. RoPE encodes offset as phase across 32 frequency planes, which downstream projections can consume as rich, directional, content-interacting structure — and it composes naturally with caching and with tricks like partial rotation and frequency rescaling. The field’s revealed preference is visible in our course model: Qwen3.8-27B ships RoPE (in the partial, long-base, multimodal form of lesson 4.7), not ALiBi, not learned buckets.

There is a deeper 2026 postscript, and it is the right closing thought for the whole positional trilogy. These three lessons implicitly assumed the architecture of 2017–2022: full attention at every layer, so the positional scheme must serve every token pair at every depth, and long-context strain lands entirely on it. Hybrid layouts break that assumption. In Qwen3.8-27B only 16 of 64 layers run full attention; the other 48 are Gated DeltaNet layers that process the sequence as an ordered recurrence — order is built into their execution, no rotation, bias, or stamp required. The positional scheme now has to be excellent in a quarter of the depth rather than all of it, while the recurrent layers carry cheap length. Debates that were fought as “which encoding extrapolates best” partly dissolve into “how much of the model needs pairwise attention at all” — the question lesson 4.15 takes up directly.

The durable picture for this family remains worth keeping: a content score plus a positional prior, position as the price of communication between two points rather than an address stamped on either. Even where ALiBi itself is not shipped, that framing — locality as a soft tariff on logits — recurs across sparse attention, sliding windows, and every scheme that must decide how much the past should cost.