Positional encoding II: RoPE

1

Concept

Absolute positional encodings attach a coordinate to each token representation. Rotary position embedding, usually shortened to RoPE, takes a different route: it rotates query and key features by angles determined by their positions. The attention dot product then carries relative displacement through geometry.

Start with a two-dimensional feature pair $(x_0,x_1)$. At position $m$, rotate it by angle $m\theta$:

RoPE applies this idea across pairs of features, using a different frequency $\theta_i$ for each pair. Some pairs rotate quickly and represent fine positional changes; others rotate slowly and cover broader ranges. The unrotated query and key still come from learned content projections. Position changes their orientation, not their norm.

The key identity is

Because rotation matrices are orthogonal, the two absolute rotations combine into one rotation by the difference $n-m$. The attention score depends on query content, key content, and their relative offset. This is why RoPE is described as encoding relative position even though each vector is rotated using its own absolute index.

In code, an implementation often splits the final feature axis into pairs or into two halves, computes sine and cosine tables, and combines them without constructing explicit matrices. The sine and cosine values must use the correct dtype, device, positions, and head dimension. A swap in pairing convention can still produce valid shapes while loading incompatible weights, so a model’s exact RoPE layout is part of its checkpoint contract.

RoPE is normally applied after projecting queries and keys and before their dot product. Values are not usually rotated. The residual stream also does not need an absolute positional vector added at the input. This keeps position tied directly to attention routing. Different heads commonly share the same frequency schedule, though architectures can vary.

Context extension methods modify how positions map to angles. Simple position interpolation compresses a longer requested range into a familiar training range. Other methods change the frequency schedule or apply dimension-dependent scaling. These interventions trade resolution at short distances against stability at long ones. They must be evaluated on both ordinary tasks and long-range retrieval; a lower perplexity at extended length does not prove every positional skill survived.

RoPE also interacts with the KV cache. During autoregressive decoding, cached keys must already carry the rotation for their original positions, while a new query receives the current position’s rotation. If position indices restart incorrectly, or cached keys are rotated twice, attention silently compares the wrong offsets. Sliding-window and chunked systems need an explicit convention for global versus local indices.

A visual explanation should show one two-dimensional plane first. Place a query arrow and a key arrow at position zero, rotate them by different position angles, and display their dot product. Then reveal that the dot product depends on the angular difference. The full embedding simply repeats that mechanism at several frequencies; it is not one giant physical rotation in three-dimensional space.

The lasting insight is that position can enter a model through an operation, not only through addition. RoPE turns sequence displacement into relative phase inside query–key similarity. That compact interface, together with its compatibility with causal attention and caching, explains why it became common in decoder-only language models.