Core

Positional encoding I: sinusoidal & learned

Because attention alone is permutation-equivariant, Transformers need an explicit signal that distinguishes position and order.

Updated 8/1/2026

Concept

Self-attention compares token representations by content. If a layer receives the same vectors in a different order and no other signal changes, its outputs are permuted in the same way. This permutation equivariance is useful for sets but insufficient for language. “Dog bites person” and “person bites dog” contain the same token identities with different roles. A Transformer therefore needs information about position.

The original architecture added a positional vector to each token embedding before the first layer:

x_t = e_{token_t}+p_t.

Because both vectors have width $d_{model}$ , their features share the residual stream. Learned projections can then use combinations of content and position when forming queries, keys, values, and feed-forward activations. Addition is cheap and preserves tensor shape, although it does not force the network to keep content and position in separable coordinates.

The 2017 sinusoidal scheme defines pairs of dimensions using different frequencies:

PE(pos,2i)=\sin\left(pos/10000^{2i/d_{model}}\right),

PE(pos,2i+1)=\cos\left(pos/10000^{2i/d_{model}}\right).

Low-index pairs change relatively quickly; higher-index pairs change slowly. The resulting vector behaves like a bank of clock hands with different periods. No two nearby positions receive the same full pattern, and relative offsets can be expressed through linear relationships between sine and cosine pairs. The values are fixed, so the model does not spend parameters on a position table.

A learned absolute position embedding uses a trainable matrix with one row per supported position. Position 37 retrieves row 37 just as a token ID retrieves a token embedding. The model can adapt those vectors to the training distribution instead of accepting a predetermined geometry. BERT and GPT-style systems have used learned absolute tables successfully.

The tradeoff is not “mathematical good versus learned good.” A learned table has a declared maximum length and no row beyond it. Extending it requires resizing or interpolation, and new rows were not trained. A sinusoidal formula can produce values for any integer position, but being defined outside the training range does not mean the rest of the network will behave well there. Extrapolation is a property of the complete trained system, not of the encoding formula alone.

Absolute positions also tie an event to an index rather than directly to a distance. To learn “look three tokens back,” a layer must infer that relationship from the absolute signals at the query and key. Relative-position methods instead modify attention according to offsets. RoPE rotates queries and keys so their interaction depends on relative displacement; ALiBi adds a distance-based bias. Those approaches will appear next.

Padding and packing complicate position indices. If batches are padded on the left, blindly numbering the full tensor changes the position IDs of identical content. Some models expect numbering to restart after padding. Packed documents may reset positions at document boundaries or continue globally, depending on the training contract. A mismatch between training and inference position handling can harm output even when shapes remain valid.

A useful inspection plots position on one axis and feature dimension on the other. Sinusoidal encodings show bands at several frequencies; a learned table shows whatever structure optimization discovered. The picture should not be mistaken for meaning by itself. What matters is how later projections use those directions.

The durable mental model is symmetry breaking. Content-only attention treats the sequence like labelled cards without slots. Positional encoding marks the slots so learned routing can distinguish “before,” “after,” “near,” and particular regions of the context. Sinusoidal and learned absolute schemes provide two different coordinate systems for doing that.

Explain it like I am five

Imagine loose dialogue cards from a play. The words identify what each actor says, but shuffling the cards changes the story. Sinusoidal encoding stamps every card with several clock hands turning at different speeds; nearby cards have related stamps, and long cycles distinguish broader locations. Learned embeddings instead give every seat in the script its own editable label. In both cases, content plus a position mark restores order.

Teach it back

Explain why self-attention needs position information and compare sinusoidal with learned absolute position embeddings.

Explain it in your own words

Minimum: 80 characters and 15 words. Your text stays only in this browser.

Saved only on this device.

Show a model answer

Without position information, permuting token vectors permutes the attention outputs in the same way, so the layer cannot distinguish sequences that contain the same tokens in different orders. Sinusoidal encodings deterministically assign multiple sine and cosine frequencies to every position and can be computed beyond trained positions, though extrapolation is not guaranteed. Learned absolute embeddings optimize one vector per supported position but require a fixed table and provide no built-in rule outside it.

Check your understanding

1. What order property does content-only self-attention lack?It cannot distinguish permutations without a positional signalIt always reads only the previous tokenIt has no learned parametersIt cannot process batches

Answer and explanation

It cannot distinguish permutations without a positional signal — Attention applied to an unordered set of token vectors is permutation-equivariant; position information breaks that symmetry.

2. What is one structural difference between sinusoidal and learned absolute encodings?Sinusoidal values follow a fixed formula; learned values come from a trained tableOnly learned encodings have a dimensionSinusoidal encodings require recurrenceLearned encodings cannot represent position zero

Answer and explanation

Sinusoidal values follow a fixed formula; learned values come from a trained table — The sinusoidal scheme has no position-table parameters, while learned absolute embeddings optimize a vector for each indexed position.

Complete the teach-back and answer the quiz correctly to finish this lesson.

Sources

Ashish Vaswani et al. (2017). Attention Is All You Need.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding.