Attention Mechanisms — Advanced

Why

RoPE encodes position by rotating each query and key vector within pairs of dimensions by an angle proportional to the token's absolute position, using the same geometric frequencies as sinusoidal encoding. The key algebraic property is that after rotating a query at position i and a key at position j, their dot product depends only on the difference i minus j, so the attention score is a function of relative distance even though the rotation itself uses absolute position. The learned-input-vector option describes additive learned positional embeddings, a different and now largely superseded method. The distance-penalty option describes ALiBi, which biases logits by distance rather than rotating vectors. The append-integer option is fabricated: RoPE does not add a raw position feature for projections to read; it transforms the existing query and key dimensions by rotation. Because RoPE is applied at every attention layer rather than once at the input, each layer receives fresh position information instead of relying on a signal surviving many residual connections. This combination of relative-distance behavior and cheap element-wise implementation is why RoPE dominates current large models.

Why

RoPE's rotation wavelengths are set by a base frequency, and a model trained at short context has only seen position gradients within that range, so naively feeding far longer inputs pushes rotations into regimes the model never optimized for. The direct fixes operate on the position encoding itself: position interpolation rescales position indices to fit the trained frequency range, and raising the base frequency stretches the low-frequency wavelengths so the model sees smooth gradients across a much longer window, which is how some models natively support 128K context. The larger-vocabulary option is wrong: better token compression reduces token count somewhat but does not change how positions beyond the trained range are encoded, and it cannot turn 128K tokens into 4K positions. The more-heads option is fabricated, since head count does not extend the position range. The disable-causal-mask option is nonsense and would break autoregression while doing nothing about position encoding. The correct family of solutions, including interpolation and its successors, all rescale RoPE's frequencies to generalize position beyond the training length.

Why

ALiBi uses no positional embeddings at all; instead it adds a fixed penalty to each attention logit that grows linearly with the distance between the query and key, with a different slope for each head so heads cover a range of receptive fields. Because the bias is a simple linear function of distance with nothing position-specific to learn, ALiBi generalizes naturally to sequences longer than those seen in training, which is its headline advantage over absolute schemes that have a hard length ceiling. The rotate-values option is wrong: ALiBi biases logits and does not rotate any vectors, and rotation of queries and keys is RoPE's mechanism. The learn-one-bias-per-position option describes a learned absolute scheme, exactly what ALiBi avoids. The concatenate-sinusoidal option is wrong because ALiBi neither concatenates encodings nor relies on sinusoids; it simply adds a distance penalty. The per-head slope diversity matters because it lets some heads attend locally for syntax and others globally for long-range dependencies. This cheap, extrapolation-friendly design is why ALiBi remains a valid alternative to RoPE for length generalization.

Why

Sinusoidal encoding is an absolute scheme injected once at the input, so it encodes fixed position vectors that the model optimizes for only within its trained length, and performance degrades beyond that range. Modern models favor relative-position methods such as RoPE and ALiBi that behave in terms of token distance, apply position at every layer, and extend more gracefully to longer contexts, which is why sinusoidal encoding fell out of use. The too-expensive option is wrong: sinusoids are cheap and can be precomputed, so cost was never the reason. The leaks-future-positions option is fabricated, since positional encoding does not bypass the causal mask, which independently blocks future tokens. The per-layer-learned-table option is wrong because sinusoidal encoding is parameter-free and is not a per-layer learned table. The essential reason is generalization and the shift to relative-position formulations, which give stronger and more extensible position signals. Sinusoidal encoding remains valuable conceptually because its frequency structure underlies RoPE's rotations.

Why

The KV cache stores a key vector and a value vector for every past token, in every layer, for every key-value head, so its size is two (for K and V) times the number of layers times the number of key-value heads times the head dimension times the sequence length, multiplied by the bytes per element. This grows linearly with sequence length and with depth, which is why long contexts are memory-hungry. The vocabulary-times-embedding option is wrong: the cache holds per-token keys and values, not output-projection rows, and vocabulary size is irrelevant to it. The square-of-sequence-length option is wrong because the cache stores keys and values, which are linear in sequence length; it is the transient attention score matrix, not the cache, that is quadratic. The query-heads-times-feed-forward option is fabricated, since the cache mirrors keys and values, not feed-forward activations, and it depends on key-value heads, not query heads. Understanding this formula explains why grouped-query attention, which reduces the key-value head count, is such an effective way to shrink the cache.