Positional encoding III: ALiBi & relative bias

1

Concept

Position information does not have to be added to token embeddings or encoded through rotations. It can alter the attention score itself. Relative-position bias methods say, in effect, “after judging content compatibility, adjust the score according to the distance and direction between these positions.”

Let $s_{ij}=q_i\cdot k_j/\sqrt{d_k}$ be the content score from query position $i$ to key position $j$. A relative-bias layer uses

Softmax then normalizes the adjusted scores. Because the bias is added before softmax, it changes routing probabilities while leaving the value payload unchanged. The function $b$ can be fixed or learned, exact or bucketed, shared or head-specific.

T5 uses learned relative-position biases with buckets. Small offsets can receive distinct buckets, while larger distances are grouped more coarsely. The model learns one bias per bucket and head. Bucketing reflects a useful prior: the exact difference between one and two tokens may matter more than the exact difference between 1,001 and 1,002 tokens. It also keeps the number of learned bias parameters bounded.

ALiBi, Attention with Linear Biases, uses a simpler fixed rule. In causal attention, a head receives a negative penalty that grows linearly with distance into the past:

The slope $m_h$ differs across heads. A steep slope strongly favors nearby keys; a shallow slope tolerates long-distance attention. Content can overcome the penalty when a distant key is sufficiently compatible. ALiBi does not impose a hard window; it supplies a graded recency prior.

Unlike a learned absolute table, ALiBi can compute a bias for distances beyond the training sequence length without allocating new rows. The authors demonstrated length extrapolation in their experiments, which motivated the “train short, test long” title. This should be interpreted carefully. Being mathematically defined at a longer length is necessary but not sufficient for robust long-context reasoning. Data, optimization, attention patterns, and evaluation task all matter.

Relative biases have a clean conceptual separation. Query and key projections ask “how compatible is this content?” The bias asks “how should displacement influence that compatibility?” With absolute embeddings, those factors are mixed earlier in the residual representation. Neither interface is universally superior; they present different inductive biases and implementation tradeoffs.

Direction matters. In a bidirectional encoder, offsets $-3$ and $+3$ may use different buckets because left and right context have different roles. In a causal decoder, future keys are forbidden by the mask, so only past distance remains. Padding and packed-document masks are still required; a relative bias does not prevent reading invalid tokens.

Comparing position schemes requires more than one long-sequence score. Check short-context quality, perplexity across lengths, retrieval at varied distances, sensitivity to repeated patterns, and actual memory cost. RoPE can represent relative phase with rich frequency structure; ALiBi offers monotonic head-specific recency; learned buckets can adapt flexible but finite patterns. Their failures differ.

The durable picture is a content score plus a positional prior. Rather than stamping every token with an address, relative bias changes the price of communication between positions. That makes locality visible in the attention matrix and gives the model several distance scales on which to route information.