Positional encoding II: RoPE

01 · Concept

Concept

Lesson 4.6 ended with a complaint: absolute encodings stamp “this is token 37” onto the input, when what attention usually needs is “this key is three tokens behind this query.” The concrete problem is to make the score itself — the query–key dot product — depend on relative offset, without entangling position into the residual stream and every value vector along the way. Rotary position embedding (RoPE) solves it with geometry: rotate queries and keys by position-dependent angles, and let the dot product do the subtraction.

Start in two dimensions. A feature pair $(x_0,x_1)$ at position $m$ is rotated by angle $m\theta$:

The content projection supplies the arrow; position turns it. Rotation preserves the vector’s norm, so the score statistics of lesson 4.3 survive intact. The payoff is the identity

Two absolute rotations meet in a dot product and collapse into one rotation by the difference $n-m$. Each vector is rotated by its own absolute index, yet the score depends only on content and relative displacement — exactly the interface we asked for. Values are not rotated, nothing is added to embeddings, and the mechanism composes cleanly with the causal mask and with caching. The full scheme just repeats this trick across many feature pairs at geometrically spaced frequencies $\theta_i$ derived from a base $\theta$: fast planes resolve fine offsets, slow planes cover broad ranges — the sinusoidal clock-bank idea relocated from the input to the score.

Now the reasonable-but-wrong assumption to correct: that a model rotates all of each head’s dimensions. Qwen3.8-27B does not. Its configuration sets partial_rotary_factor to 0.25, and the arithmetic is worth doing by hand. Head dimension is 256, so the rotated slice is

dimensions per head — and since each rotation plane consumes a pair of dimensions, that is $64 \div 2 = 32$ planes. The remaining $256 - 64 = 192$ dimensions of every query and key are not rotated directly by RoPE. That does not make them position-free: their hidden-state inputs may already encode position through earlier attention or recurrence. Partial rotation concentrates the direct rotary phase signal into a dedicated subspace while leaving most dimensions available for unrotated matching.

Two more numbers finish Qwen’s rotary picture. First, the frequency base: classic RoPE used $\theta = 10{,}000$; Qwen3.8-27B uses $\theta = 10{,}000{,}000$. A larger base stretches every wavelength, slowing the planes down so that positions hundreds of thousands of tokens apart still land at distinguishable phases — a prerequisite for its native context of 262,144 tokens. Second, the 32 planes are not one undifferentiated bank. The model is multimodal, and a video patch has three meaningful coordinates: when (frame), and where (row, column). Qwen’s MRoPE splits the 32 planes into interleaved sections of $[11, 11, 10]$ — check: $11 + 11 + 10 = 32$ — assigned to temporal, height, and width positions respectively. For ordinary text the three coordinates coincide with the token index; for images and video, each section rotates by its own axis, giving the same attention mechanism a native notion of two-dimensional and temporal layout. How image patches enter the token sequence at all is lesson 4.17’s story.

Beyond the native 262,144 tokens, the model card specifies extension to 1M tokens via YaRN, which rescales the frequency schedule so unfamiliar positions land in familiar phase ranges rather than extrapolating raw angles. The caution from lesson 4.6 applies with full force: a formula that produces angles at position 900,000 does not promise the trained network behaves there, which is why extension methods exist and why they are evaluated on long-range retrieval rather than assumed.

RoPE’s interaction with decode deserves one sentence of preview: cached keys keep the rotation of their original absolute position while each new query arrives with the current position’s rotation, so the relative-offset identity keeps holding as the sequence grows — one of several reasons this scheme won in decoder-only models. The cache arithmetic itself lands in lesson 7.2, and only Qwen’s 16 full-attention layers do any of this; its 48 DeltaNet layers track order recurrently, with no rotary machinery at all (lesson 4.15).

The lasting insight: position can enter a model through an operation rather than an addition. RoPE turns displacement into relative phase inside query–key similarity — and Qwen3.8-27B shows the 2026 refinements: rotate only a quarter of the head, slow the clock by three orders of magnitude, and deal the planes out across time, height, and width.