Positional Encoding: How a Model Knows Word Order

Shuffle the words of a sentence and, without positional information, a transformer produces the same set of outputs in shuffled order. Order is not in the architecture. It has to be added, and the four ways of adding it behave very differently once contexts get long.

Attention cannot see order

Take the attention computation: score every query against every key, softmax, blend the values. Permute the input rows and every score appears again, just relocated; the output is the same rows in the same permutation. The operation is permutation-equivariant. The feed-forward half is worse — it processes each position in complete isolation, so it cannot contribute order either.

So “dog bites man” and “man bites dog” would be indistinguishable. Every design below is a way of getting position into the vectors or into the scores.

Absolute schemes: sinusoidal and learned

The original transformer (Vaswani et al., 2017) added a fixed vector to each token embedding, built from sines and cosines at geometrically spaced frequencies. It needs no parameters and is defined for any position, including ones never seen in training.

GPT-2 and BERT instead learned a lookup table: one trainable vector per position, up to a maximum length. Simple, effective inside the trained range, and it has two hard failures. Position 4097 in a model trained to 4096 has no vector at all — not a degraded one, none. And the rows near the top of the table are updated by far fewer training examples than the rows near the bottom, so quality degrades before the limit is reached.

Both are absolute: they encode “this is position 12”. What a language model mostly needs is relative — “this token is three back from that one” — and absolute schemes make the model learn that relationship rather than giving it.

RoPE, derived

Rotary position embedding (Su et al., 2021) does something cleverer: it leaves the values alone and rotates the queries and keys. Split each query and key vector into 2-dimensional pairs. For the i-th pair, define a frequency θ_i. A token at position m has its pair rotated by the angle mθ_i.

Now do the algebra on one pair. Write the query pair as a complex number q and the key pair as k. Rotating by mθ is multiplication by e^imθ. The attention score contribution is the real part of the product of the rotated query and the conjugate of the rotated key:

  Re[ (q e^{i m theta}) * conj(k e^{i n theta}) ]
= Re[ q conj(k) e^{i (m - n) theta} ]

The absolute positions m and n have cancelled.
Only the offset (m - n) survives.

That is the entire idea, and it is worth restating because it is unusual: the encoding is applied absolutely, one rotation per token at its own position, but the dot product it produces depends only on the distance between the two tokens. Relative behaviour, absolute bookkeeping.

The absolute bookkeeping is what makes it practical. Each key is rotated once, when its position is known, and then cached forever — so RoPE is compatible with the KV cache and with prefix reuse. Additive relative-bias schemes, which add a learned term per query-key distance, require the bias at score time for every pair; they work, but they interact badly with cached and chunked attention implementations.

One more design choice is easy to miss and matters: the rotation is applied to queries and keys only, never to values. Position therefore affects which tokens are attended to and not what they contribute once attended to. That separation is the same one that makes keys and values distinct in the first place, and it means a token’s content survives being moved in the sequence — a property you rely on every time you paste the same document at a different offset in a prompt.

ALiBi: no positions at all

Press et al. (2021) removed positional vectors entirely. Instead, after computing raw attention scores, subtract a penalty proportional to the distance between query and key, with a different fixed slope per head: score becomes q·k − m × (distance). Nearby tokens are favoured; the penalty grows without bound, so nothing needs to be defined for “unseen” positions and the scheme extrapolates to lengths it never trained on by construction.

The cost is that the recency prior is hard-wired. A head with a steep slope simply cannot attend strongly to something 50,000 tokens back, however relevant. For tasks whose whole point is long-range retrieval, that is a ceiling rather than a bias.

Why RoPE won, and where it breaks

The last property is the one that settled it. Because position enters through a frequency, you can change the frequency. Position interpolation (Chen et al., 2023) compresses positions into the trained range by scaling; NTK-aware scaling and YaRN (Peng et al., 2023) adjust the frequency base non-uniformly so that high-frequency pairs, which carry local ordering, are disturbed less than low-frequency ones. A model trained at 4k can be adapted to a much longer context with a modest amount of fine-tuning. Neither a learned table nor ALiBi offers a comparable knob.

The failure mode to know: pushing RoPE past its trained length without any rescaling does not degrade gently. The rotations enter angle combinations the model never saw, attention scores become erratic, and output quality falls off sharply rather than gradually. That is a large part of why effective context length and advertised context length are different numbers, and why very long context windows are an engineering achievement rather than a configuration change.