RoPE, ALiBi, and long-context position

Rotary Position Embedding (RoPE) encodes position by rotating query and key vectors in two-dimensional subspaces before the dot product. The rotation angle is a function of absolute position, but the resulting attention score depends only on relative distance between tokens. This is the core mechanism: absolute encoding on the vectors, relative effect on the score.

Su et al. (2021) defined RoPE as follows. Split each query and key vector of dimension d into d/2 pairs of adjacent elements (x_0, x_1), (x_2, x_3), .... For each pair at dimension index i and position pos, apply a 2D rotation by angle $θ_i = \frac{\text{pos}}{10000^{2\frac{i}{d}}}$ :

x'_2i = x_2i * cos(θ_i) - x_(2i+1) * sin(θ_i)
x'_(2i+1) = x_2i * sin(θ_i) + x_(2i+1) * cos(θ_i)

The frequency formula is identical to sinusoidal PE — the insight is applying it as a rotation to Q and K rather than as an addition to the input embedding. Critically, RoPE is applied at every attention layer, not just at the input. This gives each layer fresh position information, rather than relying on position signals surviving through dozens of residual connections (the weakness of additive positional encoding).

Why rotation produces relative position

The dot product between two rotated vectors has a clean algebraic property. For query at position m and key at position n, each 2D subspace contributes:

q_{\text{rotated}} \cdot k_{\text{rotated}} = (q \cdot k) \cdot \cos((m - n) \cdot θ_i) + \text{cross}\_\text{term} \cdot \sin((m - n) \cdot θ_i)

The angles m * θ_i and n * θ_i cancel to (m - n) * θ_i. The attention score depends on the relative distance m - n, not on the absolute positions m and n individually. A token at position 5 attending to position 3 produces the same positional contribution as position 105 attending to position 103.

python

import numpy as np

def apply_rope(x, positions, d_model):
    """Apply RoPE to vectors x at given positions.
    x: (seq_len, d_model), positions: (seq_len,)
    """
    out = np.zeros_like(x)
    for i in range(d_model // 2):
        theta = positions / (10000.0 ** (2 * i / d_model))  # (seq_len,)
        cos_t = np.cos(theta)
        sin_t = np.sin(theta)
        out[:, 2*i]     = x[:, 2*i] * cos_t - x[:, 2*i+1] * sin_t
        out[:, 2*i + 1] = x[:, 2*i] * sin_t + x[:, 2*i+1] * cos_t
    return out

d = 64
seq_len = 128
positions = np.arange(seq_len).astype(float)

np.random.seed(42)
q = np.random.randn(seq_len, d)
k = np.random.randn(seq_len, d)

q_rot = apply_rope(q, positions, d)
k_rot = apply_rope(k, positions, d)

# Dot product between position 5 and 3 (distance = 2)
score_5_3 = q_rot[5] @ k_rot[3]
# Dot product between position 105 and 103 (distance = 2)
score_105_103 = q_rot[105] @ k_rot[103]

print(f"Score(pos=5, pos=3):     {score_5_3:.4f}")
print(f"Score(pos=105, pos=103): {score_105_103:.4f}")

To isolate the positional component, set q and k to the same constant vector:

python

q_const = np.ones((seq_len, d))
k_const = np.ones((seq_len, d))
q_rot = apply_rope(q_const, positions, d)
k_rot = apply_rope(k_const, positions, d)

# Now the dot product is purely positional
print(f"Positional score, distance=2, pos=(5,3):     {q_rot[5] @ k_rot[3]:.4f}")
print(f"Positional score, distance=2, pos=(105,103): {q_rot[105] @ k_rot[103]:.4f}")
# These are exactly equal — proving relative-position dependence

RoPE and context length extension

RoPE's base frequency of 10000 determines the wavelength of each rotation dimension. For a model trained at context length 4096, the highest-frequency dimension completes one full rotation every ~6 positions, and the lowest-frequency dimension completes one rotation over the entire context. Position interpolation (Chen et al. 2023) scales the position index down to fit a longer context into the trained frequency range:

θ_i = \frac{(\frac{\text{pos}}{\text{scale}})}{10000^{2\frac{i}{d}}}

where $\text{scale} = \frac{\text{target}\_\text{length}}{\text{trained}\_\text{length}}$ .

To extend from 4096 to 32768 tokens, set scale = 8. Position 32768 maps to the same rotation angle as position 4096 in the original encoding. This compresses the position information but preserves the relative distance property. Meta applied this technique to extend Llama 2's context from 4K to up to 32K tokens with only 1000 steps of fine-tuning.

YaRN (Peng et al. 2023) improved on linear interpolation with NTK-aware scaling. Instead of uniformly scaling all frequency dimensions, YaRN applies stronger scaling to low-frequency dimensions (which contribute most to long-range position discrimination) and weaker scaling to high-frequency dimensions (which are already fine-grained). Combined with a temperature adjustment to the attention logits, YaRN extends Llama 2 7B from 4K to 128K context with less than 0.1 perplexity increase on the original context range.

Llama 3 (Meta 2024) increased RoPE's base frequency from 10000 to 500000, which stretches the low-frequency wavelengths by 50x. This natively supports 128K context without any interpolation — the rotation periods are long enough that the model sees smooth position gradients across the full window.

ALiBi: attention with linear biases

Press et al. (2022) took a different approach: no positional encoding at all. ALiBi adds a fixed linear penalty to attention scores based on token distance. For attention head h with slope m_h, the bias applied to the attention logit between query at position i and key at position j is:

\text{bias} = - m_{h} \cdot |\text{i - j}|

Slopes are set as a geometric sequence: for 8 heads, $m = [\frac{1}{1}, \frac{1}{2}, \frac{1}{4}, \frac{1}{8}, \frac{1}{16}, \frac{1}{32}, \frac{1}{64}, \frac{1}{128}]$ . Some heads have steep slopes (strongly penalizing distant tokens, focusing locally), while others have gentle slopes (allowing long-range attention). The slopes are fixed — never learned or updated during training.

python

import numpy as np

def alibi_bias(seq_len, num_heads):
    """Compute ALiBi attention bias matrix for each head."""
    ratio = 2 ** (-8 / num_heads)
    slopes = np.array([ratio ** (i + 1) for i in range(num_heads)])

    positions = np.arange(seq_len)
    distances = np.abs(positions[:, None] - positions[None, :])  # (seq_len, seq_len)

    bias = -slopes[:, None, None] * distances[None, :, :]  # (num_heads, seq_len, seq_len)
    return bias

bias = alibi_bias(seq_len=8, num_heads=4)
print(f"Head 0 (steep slope): bias[0,7] = {bias[0, 0, 7]:.4f}")
print(f"Head 3 (gentle slope): bias[3,0,7] = {bias[3, 0, 7]:.4f}")

ALiBi is computationally cheap: the bias matrix is precomputed once per sequence length and added to the attention logits before softmax. No additional parameters, no rotation, no learned embeddings. The bias is causal-compatible — for autoregressive models, only the lower-triangular portion of the bias matrix is used, matching the causal attention mask.

ALiBi's key advantage is extrapolation. Since the bias is a simple linear function of distance, it generalizes naturally to positions beyond the training range — there is nothing position-specific to learn. Press et al. showed that a 1024-context ALiBi model maintains performance up to 2048 tokens with no fine-tuning, and degrades gracefully beyond that. The slope diversity across heads is essential: if all heads had the same slope, the model could not simultaneously attend locally (for syntax) and globally (for long-range dependencies). The geometric spacing ensures a range of receptive fields without any tuning.

Which models use what

RoPE is the dominant positional encoding in the current generation of large language models. Llama 3 (8B, 70B, 405B), Mistral (7B, 8x7B, 8x22B), Qwen 2.5, DeepSeek-V2, and Gemma 2 all use RoPE. The technique has won because it combines relative-position attention with simple and efficient implementation — the rotation is applied as element-wise multiplications and additions, with no additional parameters.

ALiBi is used by Falcon (7B, 40B, 180B), MPT (7B, 30B), and BLOOM (176B). It remains a valid choice, especially for applications that need extrapolation beyond training context without any fine-tuning.

Sinusoidal positional encoding, as proposed in the original transformer paper, is no longer used in any major LLM. Its last notable use was in the original T5 (Raffel et al. 2020), which used a learned relative position bias — itself a predecessor to ALiBi. For any new model architecture in 2026, the choice is effectively between RoPE and ALiBi, with RoPE being the default.

Implementation cost

RoPE adds negligible overhead: the rotation is four element-wise multiplications and two additions per dimension pair, applied to Q and K at every layer. For Llama 3 8B with $d_{\text{head}} = 128$ , each head requires 64 rotation operations per token — trivial compared to the 128 * 128 matrix multiplication for the Q/K/V projections. The rotation angles can be precomputed for all positions up to max_len and stored as a (max_len, d_head) buffer of cosines and sines.

ALiBi is even cheaper: the bias matrix is O(seq_len^2) values, computed once per forward pass (or once per sequence length and cached). It adds a single element-wise addition to the (num_heads, seq_len, seq_len) attention logit tensor.

Neither method introduces learnable parameters. Both are deterministic functions of position — no gradient updates, no optimizer state, no risk of overfitting to position patterns.

← Previous