Self-attention begins as an unusually polite meeting.

Give it a set of token vectors and, before positional information is added, it does not know which token came first. The sequence is present as memory layout, but not as a fact in the dot products.

The original Transformer solved this by adding sinusoidal positional encodings to token embeddings.1 Later architectures tried many variants: learned absolute embeddings, relative position biases, linear distance biases, and rotary embeddings.

RoPE, the rotary position embedding from RoFormer, is one of the more elegant answers because it does not add position as another vector. It winds each query/key plane like a clock hand.2

not: token + position
but: rotate query and key by position

That small change buys the identity that makes the whole trick feel like a magic trick you are allowed to inspect.

One Plane, One Clock Hand

Take one pair of coordinates, \((x_0, x_1)\). A position \(m\) rotates that pair by angle \(m\theta\):

\[R_{m\theta} \begin{bmatrix} x_0 \\ x_1 \end{bmatrix}.\]

The full embedding is split into many such two-dimensional planes. Each plane gets its own frequency \(\theta_i\). In the common RoPE schedule,

\[\theta_i = B^{-2i/d},\]

where \(B\) is a base such as 10000, \(d\) is the head dimension, and \(i\) is the coordinate-pair index.

Now rotate query \(q\) at position \(m\) and key \(k\) at position \(n\). Because rotation matrices are orthogonal,

\[(R_m q)^\top (R_n k) = q^\top R_{n-m} k.\]

The absolute positions have not vanished; they have cancelled into a relative phase. This is the core RoPE trick: the attention score can depend explicitly on the displacement \(n-m\) while still using ordinary dot-product attention.

The lab below verifies that identity numerically with random query and key vectors. In the default run, the relative-identity error is 1.57e-13, which is just floating-point noise.

Microscope for Phase

The lab uses a simple diagnostic kernel:

\[K(\Delta) = \frac{1}{d/2}\sum_i \cos(\Delta\theta_i).\]

This is not a language-model perplexity curve. It is what the positional part does to two identical pieces of content separated by \(\Delta\) tokens when all frequency planes are weighted equally. That makes it a clean microscope for phase, aliasing, and interpolation.

Raw extrapolation Position interpolation Training length Focus frequency

The lab plots a phase-only diagnostic, not a trained model's attention distribution. Position interpolation maps target positions back into the trained range by multiplying positions by train length divided by target length.

In the default setting, train length is 2,048 and target length is 8,192, so position interpolation uses scale 0.25.

The local effect is immediate:

raw one-token kernel drop:           0.03385
interpolated one-token kernel drop:  0.00222

Interpolation keeps target positions inside the old phase range, but it also compresses nearby positions. The clock ticks more slowly everywhere.

I also swept 243 combinations of head dimension, base, train length, target length, and seed. The largest relative-identity error I saw was about 1.8e-12, still numerical dust. The diagnostic is not testing a trained model; it is checking that the rotation algebra and the displayed knobs agree.

That is the trade:

extrapolation asks the model to interpret unseen phases
interpolation asks the model to read with a compressed ruler

Neither sentence says whether a particular long-context model will work. That depends on training, finetuning, data, attention heads, retrieval behavior, evaluation task, and many implementation details. The lab is narrower: it shows what the position map itself is doing.

A Longer Window Retunes the Clocks

RoPE helped make position handling feel smooth because it gives every pair of coordinates a clean relative phase story. But a frequency ladder is still a frequency ladder.

At position \(p\), pair \(i\) sees angle:

\[p\theta_i.\]

If inference goes far beyond the training positions, some frequency planes are being evaluated at phase regimes the model never had to use during training. Because phases wrap, distant positions can also become deceptively similar in some planes. The whole embedding is multi-frequency, so there is no single period, but the diagnostic kernel still oscillates.

This is why context-extension methods are not only memory-engineering tricks. They are positional-distribution interventions.

Position Interpolation, for example, linearly down-scales input position indices so a longer target window maps into the original trained context window.3 It is attractive because it avoids raw extrapolation into much larger positions. The price is visible in the lab’s local-resolution panel: nearby tokens become less phase-separated unless further training adapts the model.

ALiBi takes a different route. It does not rotate or add embeddings; it adds a linear distance penalty to attention scores and was proposed with length extrapolation in mind.4 That inductive bias says something very explicit: farther keys should usually be penalized. RoPE says something subtler: relative position is a collection of phases.

Not a Coordinate Sticker

The bad mental model is:

RoPE tells the model the absolute position

It is not completely false. The rotation angle uses the absolute position before the dot product. But the reason RoPE is powerful in attention is that the query-key score collapses those absolute rotations into relative rotation.

The better mental model is:

RoPE gives each attention head a bank of clocks
attention reads relative phase differences between clocks

The clocks have different speeds. Fast clocks distinguish local offsets but wrap quickly. Slow clocks carry coarser distance information over longer spans. The model learns how to use that bank while solving its training tasks.

When we extend context, change the base, interpolate positions, or rescale frequencies, we are changing the clock bank. Sometimes that is exactly what we want. It should not be mistaken for a harmless metadata edit.

Before I Trust the Window

Before trusting a RoPE context-extension claim, I would want more than a maximum context number.

I would check:

  1. the original training context and the target context;
  2. the exact frequency schedule and any scaling rule;
  3. local tasks that depend on neighboring tokens;
  4. long-range retrieval tasks such as passkey-style probes;
  5. tasks with many distractors, not only one planted fact;
  6. attention/logit stability across positions;
  7. whether finetuning data actually exercised the new range.

Long context is not one capability. It is a family of position-sensitive behaviors.

RoPE makes the position mechanism unusually beautiful. It also makes the core question unusually concrete:

which phases did the model learn to read?

  1. Ashish Vaswani et al., “Attention Is All You Need”, NeurIPS 2017. The paper introduced the Transformer architecture and used sinusoidal positional encodings in the original model. ↩

  2. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, and Yunfeng Liu, “RoFormer: Enhanced Transformer with Rotary Position Embedding”, arXiv 2021. ↩

  3. Shouyuan Chen, Sherman Wong, Liangjian Chen, and Yuandong Tian, “Extending Context Window of Large Language Models via Positional Interpolation”, arXiv 2023. ↩

  4. Ofir Press, Noah A. Smith, and Mike Lewis, “Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation”, ICLR 2022. ↩