Attention Mechanisms — Core

Why

A recurrent network folds everything it has seen into a single fixed-size hidden state and updates it one step at a time, so information from an early token must survive many nonlinear updates to influence a later one, and gradients through those updates shrink, causing the vanishing-gradient problem. This both limits how far dependencies can reach and forces strictly sequential computation that cannot use the parallelism of GPUs across the sequence. Attention fixes this by letting any position read directly from any other position in a single step, giving constant path length and full parallelism. The too-many-parameters option misdiagnoses the problem: parameter count was not the core issue, and RNNs already share weights across timesteps. The fixed-length-input option is wrong, since RNNs handle variable-length sequences natively. The labeled-alignment option is wrong because RNNs do not require token-pair alignment labels, and attention weights are learned, not supervised. The real breakthrough was direct, learned all-to-all connections that remove the sequential bottleneck.

Why

Attention projects each token into a query, a key, and a value. The query vector encodes what the token is looking for, the key vector encodes what a token offers as a match, and their dot product measures compatibility, so a high query-key score means one token should attend to another. The output-and-gradient option is wrong: the query is an input-side projection used to compute scores, not an output or a gradient. The identical-copies option is wrong and misses the whole point of separating Q and K, which lets attention be asymmetric so token A can attend to B without B attending equally to A. The position-and-embedding option confuses query and key with positional encoding; both Q and K are content projections, and position is injected separately. Keeping query and key as distinct learned projections is what lets the model represent directional relationships, such as a verb attending to its subject. The value projection then carries the actual content transmitted when a token is attended to.

Why

A dot product between a query and key is a sum of d_k component products, so its variance grows with d_k and the raw scores spread to large magnitudes as the head dimension increases. Feeding large logits into softmax pushes it toward a near one-hot distribution where gradients are almost zero, which stalls learning. Dividing by the square root of d_k rescales the scores back to roughly unit variance, keeping softmax in a responsive, high-gradient range. The convert-to-probabilities option is wrong because softmax, not the scaling constant, produces the probability distribution; scaling only adjusts magnitudes. The reduce-parameters option is wrong: the scale is a fixed constant, not a learned parameter, and it changes no matrix shapes. The always-uniform option is wrong because scaling does not force uniform weights; the model still learns sharp, content-dependent attention. This variance-control trick is why dot-product attention needs the scale while additive attention, bounded by a tanh, does not.

Why

The softmax turns each token's scores into a probability distribution over positions, and the output for that token is the weighted sum of all value vectors using those probabilities as weights. So the output blends content from across the sequence in proportion to how well each key matched the query. The single-value option is wrong because attention is soft: even the top position rarely gets all the weight, and the output mixes multiple values rather than copying one. The query-times-key option is wrong because the values, not the queries and keys, form the output; Q and K only determine the weights. The concatenate-keys option is wrong because keys are used to score, not to build the output, and there is no such concatenation step. This weighted-average-of-values structure is what lets a token pull in information from wherever it attends, and the sharpness of the distribution controls how focused that mixing is. In a trained model these distributions are often concentrated on a few positions.

Why

A single attention head learns one projection of queries and keys, which defines one notion of which tokens are related, so it cannot simultaneously capture, say, subject-verb agreement and long-range coreference. Multi-head attention runs several attention operations in parallel, each with its own projections, so different heads can specialize in different relationship types, and these roles emerge from training rather than being assigned. The increase-compute option is wrong: the model dimension is split across heads, so multi-head attention costs about the same as one head on the full dimension. The redundant-backups option is wrong because heads do not store copies of the sequence; they compute different views of the relationships. The remove-positional-encoding option is wrong because heads do not encode order on their own, and position must still be injected explicitly. The payoff of multiple heads is representational: several distinct similarity metrics operating at once, later recombined by the output projection.