What is rotary position embedding (RoPE)?
Rotary position embedding (RoPE) tells a transformer where each token sits in a sequence by rotating its query and key vectors before the attention dot product, instead of adding a separate position signal to the token's representation. Each vector is rotated by an angle proportional to its position. The dot product of two rotated vectors depends only on the angle between them, so the attention score depends on the relative distance between two tokens, not their absolute positions.
The problem position embeddings solve
Self-attention compares tokens with no built-in sense of order: shuffle a sequence and, without position information, the attention score between any pair stays the same. Early transformers fixed this by adding a fixed or learned position vector to each token's embedding before the first layer, mixing absolute position into the token representation.
What RoPE does differently
RoPE rotates the query and key vectors directly. It splits each vector into pairs of dimensions and rotates each pair by an angle that grows with the token's position, with a different rotation frequency per pair. As a result, the dot product between a query at position i and a key at position j is a function of i minus j, not of i and j separately. Two tokens five positions apart give the same relative signal at the start of the sequence or near the end, which is more useful to attention than raw absolute position.
Why relative position, not absolute, is what attention needs
For most language patterns, what a token needs to know about another is how far away it is and in which direction, not which absolute index either occupies. A verb needs to find its subject a few tokens back whether the pair sits at token 10 or token 10,000. RoPE puts relative distance straight into the attention score instead of making the model infer it from two absolute signals, and that is why most current open-weight LLMs use it in place of added position vectors.
How it connects to context length
RoPE's rotation frequencies are tuned to the range of positions seen in training. Running a model well beyond that range can degrade attention quality unless the frequencies are rescaled, an adjustment usually called RoPE scaling or context extension. That is a separate concern from the KV cache memory a longer sequence needs, covered in the KV cache glossary entry, and from the context window post's coverage of what a stated context length means for a given model. See handling long context requests for what this looks like from the serving side, including on a Spark's 128 GB of unified memory, where the practical limit on context length is usually memory for the KV cache rather than the position encoding scheme itself.