Positional encoding — injecting order into attention

Attention cannot tell you what order it read

Go back to the dot products from lesson 2. Every score was computed from a pair of vectors, and at no point did the position of either token enter the arithmetic. Shuffle the tokens and the same pairs produce the same scores; the softmax over a reordered set of the same numbers gives the same weights to the same tokens. Self-attention is permutation-equivariant — reorder the input and you get the identically reordered output, never a different answer.

That is a catastrophe for language.

"the server rejected the request" and "the request rejected the server" contain exactly the same tokens. To a bare attention layer they are the same input, and it will produce the same representations for both. An RNN never had this problem — order was implicit in the fact that it read left to right. Removing recurrence bought parallelism and lost sequence, and the loss has to be paid back explicitly.

The fix is almost crude: before the first layer, add a vector to each token embedding that depends only on the position. Position 0 gets one signature, position 1 gets another. Since the signature is added into the same vector, it flows through the Q/K/V projections along with the meaning, and the dot products start depending on where things are.

Why sine and cosine

The signature could be anything distinct — even the position number in one dimension. Sinusoids earn their place for a specific reason: they make relative distance readable.

Look at the table. Dimensions 0 and 1 are a sine/cosine pair at wavelength 2π; dimensions 2 and 3 are the same pair at 100× the wavelength. The fast pair separates neighbours crisply — positions 1 and 2 differ by 0.8415 versus 0.9093 — but it wraps around, so on its own it cannot distinguish position 1 from a position one full cycle later. The slow pair barely moves between neighbours (0.0100 to 0.0200) but never repeats over any realistic length. Read together, the pair fixes a position exactly, the way an hour hand and a minute hand do.

The deeper property is that this scheme is linear in offset: the encoding at position pos + k can be written as a fixed rotation of the encoding at pos, where the rotation depends on k alone and not on pos. So "the token four places back" is a consistent geometric relationship the model can learn once and apply everywhere — including at sequence lengths longer than anything it was trained on, since the formula is defined for every position rather than looked up from a learned table.

Whether that extrapolation actually works in practice is a separate question, and the answer is mostly no — which is why RoPE and ALiBi exist, and why the next lesson is about them.

Self-attention computes a weighted sum over all input positions. The operation is permutation-invariant: given a set of input vectors, shuffling their order and then permuting the output the same way produces an identical result. Formally, for any permutation matrix P, $\text{Attention}(\text{PX}) = P \cdot \text{Attention}(X)$. The model has no mechanism to distinguish position 0 from position 47 — unless position information is explicitly added to the input.

Every production transformer injects position before the first attention layer. The methods differ, but the constraint is universal: without positional encoding, a transformer treats "the cat sat on the mat" and "mat the on sat cat the" as identical inputs.

Sinusoidal positional encoding

Vaswani et al. (2017) proposed a deterministic encoding using sine and cosine waves at geometrically spaced frequencies. For a token at position pos in a model with embedding dimension d_model, the encoding is:

• $$\text{PE}(\text{pos}, 2i) = \sin(\frac{\text{pos}}{10000^{2\frac{i}{d_{\text{model}}}}})$$
• $$\text{PE}(\text{pos}, 2i + 1) = \cos(\frac{\text{pos}}{10000^{2\frac{i}{d_{\text{model}}}}})$$

Each pair of dimensions (2i, 2i+1) forms a sinusoidal wave with wavelength 2π * 10000^(2i / d_model). Dimension pair 0 has wavelength 2π ≈ 6.28 (oscillates rapidly across positions). The last dimension pair has wavelength 2π * 10000 ≈ 62,832 (nearly flat across typical sequence lengths). The result is a unique fingerprint for each position — a vector of 512 values (for $d_{\text{model}} = 512$) that encodes where a token sits in the sequence.

The encoding is added element-wise to the token embedding before the first transformer block: input = token_embedding + positional_encoding. Both are vectors in R^d_model, so addition is well-defined. The model learns to disentangle content from position during training.

Why sine and cosine together

Each dimension pair (sin, cos) at frequency i traces a circle in 2D as position advances. A linear transformation can convert any absolute position encoding to a relative one: PE(pos + k) can be expressed as a linear function of PE(pos) for any fixed offset k. Vaswani et al. hypothesized this would help the model learn relative position patterns. The dot product between two positional encodings, PE(pos) · PE(pos + k), depends only on k (the offset), not on pos itself — a form of translation invariance.

Properties and limitations

The dot product between position vectors decays smoothly with distance. Nearby positions have high similarity; distant positions are nearly orthogonal. This gives the model a built-in inductive bias: tokens near each other are "closer" in the position-augmented embedding space.

The encoding is also unique for every position within practical sequence lengths. Because the frequencies are geometrically spaced, no two positions produce the same combination of sine and cosine values across all d_model/2 frequency bands. The encoding can be precomputed once and stored as a constant matrix — no gradient updates, no learned parameters, no memory overhead beyond the (max_len, d_model) buffer.

But sinusoidal encoding is absolute — it encodes position 0, 1, 2, ... as fixed vectors. A model trained with max_len=512 has never seen PE(513). While the encoding function can generate values for any position, the model hasn't learned to use them. In practice, performance degrades rapidly beyond the trained context length. The degradation is not graceful: attention patterns become erratic within a few positions past the training limit, because the model's learned weights assume the position signal stays within the distribution it saw during training.

Learned positional embeddings

GPT-2 (Radford et al. 2019) replaced the deterministic formula with a learnable embedding table. A matrix of shape (max_positions, d_model) — for GPT-2, (1024, 768) — is initialized randomly and trained alongside the rest of the model. Position t is encoded by looking up row t of this table, and the result is added to the token embedding just like the sinusoidal case.

import torch
import torch.nn as nn

class LearnedPositionalEmbedding(nn.Module):
    def __init__(self, max_positions, d_model):
        super().__init__()
        self.embedding = nn.Embedding(max_positions, d_model)

    def forward(self, seq_len):
        positions = torch.arange(seq_len)
        return self.embedding(positions)  # shape: (seq_len, d_model)

pe = LearnedPositionalEmbedding(max_positions=1024, d_model=768)
pos_vectors = pe(seq_len=128)  # (128, 768)

The advantage over sinusoidal: the model can learn arbitrary position representations rather than being constrained to a fixed functional form. In practice, the two perform comparably on tasks within the trained context length (Vaswani et al. found no significant difference). The disadvantage is identical: both are absolute and fixed-length. GPT-2's 1024-position table cannot represent position 1025 — the lookup index is out of bounds.

BERT (Devlin et al. 2019) also uses learned positional embeddings, with max_positions=512. This is why BERT cannot process inputs longer than 512 tokens without architectural modification.

Parameter cost

A learned position embedding table is small relative to the model. GPT-2's (1024, 768) table contains 786,432 parameters — 0.5% of the model's 124M total. The original transformer's sinusoidal encoding uses zero parameters (it is a deterministic function). Neither approach dominates on parameter efficiency; both are negligible at scale. The choice between them comes down to flexibility (learned) versus guaranteed mathematical properties (sinusoidal).

Implementing sinusoidal positional encoding

import numpy as np
import torch

def sinusoidal_pe(max_len, d_model):
    pe = np.zeros((max_len, d_model))
    position = np.arange(max_len)[:, np.newaxis]  # (max_len, 1)
    div_term = 10000.0 ** (np.arange(0, d_model, 2) / d_model)  # (d_model/2,)
    pe[:, 0::2] = np.sin(position / div_term)  # even dimensions
    pe[:, 1::2] = np.cos(position / div_term)  # odd dimensions
    return pe

pe = sinusoidal_pe(max_len=128, d_model=512)
# pe.shape: (128, 512)

# Verify: dot product depends on offset, not absolute position
dot_1_2 = pe[1] @ pe[2]
dot_50_51 = pe[50] @ pe[51]
dot_100_101 = pe[100] @ pe[101]
print(f"PE(1)·PE(2)   = {dot_1_2:.4f}")
print(f"PE(50)·PE(51) = {dot_50_51:.4f}")
print(f"PE(100)·PE(101) = {dot_100_101:.4f}")
# All three are approximately equal — dot product depends on offset (1), not position

The output confirms translation invariance: PE(1)·PE(2) ≈ PE(50)·PE(51) ≈ PE(100)·PE(101) ≈ 238.5 (for d_model=512). The dot product encodes distance, not location.

Visualizing the wave structure

import matplotlib.pyplot as plt

pe = sinusoidal_pe(max_len=128, d_model=512)

fig, axes = plt.subplots(4, 1, figsize=(12, 8))
dims = [0, 1, 50, 100]
for ax, d in zip(axes, dims):
    ax.plot(pe[:, d])
    ax.set_ylabel(f"dim {d}")
    ax.set_xlim(0, 127)
axes[-1].set_xlabel("Position")
axes[0].set_title("Sinusoidal PE: different dimensions = different frequencies")
plt.tight_layout()
plt.savefig("sinusoidal_pe_waves.png", dpi=150)

Dimension 0 (sin) oscillates rapidly — period of about 6 positions. Dimension 100 oscillates slowly — period of about 197 positions. The model reads position from this spectrum of frequencies, the same way a Fourier decomposition encodes a signal across frequency bands.

A useful way to see the full pattern is as a heatmap of pe[:, :] — position on the x-axis, dimension on the y-axis. The result looks like a set of interference fringes: rapid vertical stripes in the low dimensions, gradually flattening to near-constant in the high dimensions. Each row of the heatmap is one position's full encoding vector. The fact that no two rows are identical (within practical sequence lengths) is what makes the encoding useful — the model can distinguish any two positions by reading different subsets of dimensions.

Numerical precision

At very high positions (>10,000) and low frequency dimensions, the argument to sin and cos becomes extremely small — pos / 10000^(2i/d) approaches zero for large i. This means the high-frequency dimensions encode position clearly, while the lowest-frequency dimensions are nearly constant across short sequences. For $d_{\text{model}} = 512$, the last dimension pair has wavelength ~62,832. Over a 512-token sequence, this dimension traverses less than 3% of one period — contributing almost no positional discrimination. The redundancy is intentional: those dimensions become useful only for very long sequences (thousands of tokens), where the high-frequency dimensions have wrapped around multiple times and become ambiguous on their own.

How positional encoding integrates with the transformer

In the original transformer architecture (and GPT-2, BERT, and most models through ~2021), positional encoding is applied once, before the first layer:

class TransformerInput(nn.Module):
    def __init__(self, vocab_size, d_model, max_len):
        super().__init__()
        self.token_emb = nn.Embedding(vocab_size, d_model)
        self.pos_enc = sinusoidal_pe(max_len, d_model)  # or nn.Embedding

    def forward(self, token_ids):
        # token_ids: (batch, seq_len)
        seq_len = token_ids.shape[1]
        x = self.token_emb(token_ids)  # (batch, seq_len, d_model)
        x = x + torch.tensor(self.pos_enc[:seq_len], dtype=x.dtype, device=x.device)
        return x  # fed into the first transformer block

Position information enters through addition and must survive through residual connections across all layers. By the final layer, the model has had many opportunities to transform the position signal — but it was injected only once, at the bottom. This is a design choice, not a necessity. Later methods (RoPE, ALiBi) inject position information at every attention layer, which gives the model stronger position signal throughout the network.

The context length ceiling

Both sinusoidal and learned positional encodings impose a hard context length limit. The model is trained with positions 0 through max_len - 1. At inference time, any input longer than max_len either raises an index error (learned embeddings) or produces untrained representations (sinusoidal — the math works but the model has never optimized for those values).

GPT-2: 1024 tokens. BERT: 512 tokens. The original transformer: 512 tokens. These limits were architectural constraints, not hardware limits.

Several workarounds exist for stretching absolute position embeddings beyond their trained range. Interpolation (scaling position indices to fit within max_len) and extrapolation (hoping the model generalizes) both degrade quality rapidly. Fine-tuning on longer sequences with extended position tables works but is expensive — it requires training data at the new length and may destabilize other learned behaviors. Breaking past fixed context length required fundamentally rethinking how position is encoded — which is the subject of the next lesson.