Transformer Architecture

Why

Self-attention allows each token in a sequence to compute a weighted combination of all other tokens, where the weights reflect learned relevance scores derived from the content itself. This means every token's output representation is context-dependent, incorporating information from anywhere in the sequence in a single computational step. The relevance weights are produced by comparing Query and Key vectors through dot products, making them dynamic and input-specific rather than fixed or predetermined. This content-based weighting is the fundamental distinction between attention and older, more rigid aggregation methods. Option A describes the fixed-length context vector bottleneck of early sequence-to-sequence models, which was precisely the limitation that attention mechanisms were invented to overcome, since compressing an entire sentence into one vector loses information. Option B describes a convolution, where the kernel weights are fixed parameters learned during training and applied uniformly regardless of input content, covering only a local neighborhood rather than the full sequence. Option D describes recurrent processing, where a hidden state is passed sequentially from one token to the next, creating the serial dependency that makes RNNs slow to train and prone to forgetting distant context. The fully parallel, content-dependent nature of self-attention is what enables Transformers to be trained efficiently on modern hardware while capturing long-range dependencies. This mechanism is the foundation of virtually every modern large language model and understanding it is essential for grasping how these models represent and manipulate meaning.

Why

In the attention mechanism, the Query vector represents what information a token is searching for, the Key vector represents what information each token has to offer, and the Value vector holds the actual content to be retrieved and blended. The dot product between a Query and all Keys produces relevance scores, and these scores, after softmax normalization, determine how much of each Value vector is included in the output for that position. All three are produced by separate learned linear projections of the same input embedding, meaning each projection can learn to extract different aspects of the token for different roles. This separation is what gives attention its expressiveness, because the criteria for matching (Query against Key) are decoupled from the information being passed forward (Value). Option A incorrectly claims that Keys and Values hold positional encodings, but positional information is added to the input embeddings before the Q, K, V projections are applied and does not occupy its own separate stream. Option C confuses attention with supervised learning by suggesting Values store target output labels, when in fact Values are simply projected representations of the input tokens that get mixed according to the computed attention weights. Option D claims all three projections are identical copies for numerical redundancy, but if they were identical, the mechanism would lose its ability to independently control what to search for, what to match against, and what content to retrieve. This Query-Key-Value decomposition was inspired by information retrieval systems, where a search query is matched against database keys to retrieve corresponding values. Understanding this three-way split is essential for grasping more advanced variants like grouped-query attention and multi-query attention, which modify how Keys and Values are shared across Query heads.

Why

When the key dimension d_k is large, the dot products between Query and Key vectors tend to grow in magnitude because the variance of the sum scales with the number of dimensions being summed. These large-magnitude scores push the softmax function into regions where its output is nearly one-hot, meaning gradients become vanishingly small and the model struggles to learn nuanced attention patterns. Dividing by the square root of d_k rescales the dot products back to unit variance, keeping the softmax in a regime where gradients flow healthily and the model can learn to distribute attention across multiple positions. This scaling factor is derived from the statistical observation that the variance of a dot product of two random vectors with d_k independent components is proportional to d_k. Option A is wrong because the scaling is applied to the scores (the Q-K dot products), not to the Value vectors, and it has nothing to do with making Values unit length. Option B confuses the roles: the softmax itself is what makes the attention weights sum to one, and that summation property holds regardless of whether the scores are scaled or not. Option C is incorrect because dividing a scalar into the scores does not change the number of parameters in any projection matrix; the projection dimensions remain the same. This scaling trick, often called scaled dot-product attention, is simple but critical for stable training and is used in virtually every Transformer implementation. Without it, deeper models with larger hidden dimensions would be especially prone to training instability due to the softmax saturation problem.

Why

For a single query, the softmax is applied across all key positions, converting the raw dot-product scores into a probability distribution over the sequence. This means the resulting attention weights are non-negative and sum to one across the key dimension, ensuring that the output for each query is a proper weighted average of the Value vectors. The key insight is that each query independently forms its own distribution over the keys, deciding how much to attend to every other position in the sequence. This per-query normalization is what allows different positions to attend to different parts of the input simultaneously. Option B, normalizing across the feature dimension within a single key vector, would not produce a meaningful distribution over positions and would instead distort the internal representation of each key. Option C, normalizing across the batch, would create bizarre dependencies between unrelated examples in a mini-batch, coupling the attention patterns of one sentence to those of a completely different one. Option D, normalizing across the value dimension after the weighted sum, would alter the output representation rather than controlling how Values are mixed, and it would come too late to serve as an attention weighting. This axis choice is fundamental to how attention operates as a soft retrieval mechanism, selecting from all available positions proportionally to their relevance. Getting the softmax axis wrong in an implementation is a subtle bug that can produce plausible-looking but completely incorrect attention patterns.

Why

Once the attention weights are computed via the softmax of the Query-Key scores, the output for each position is produced by taking the weighted sum of all Value vectors, where each Value is scaled by its corresponding attention weight. This produces a soft blend of information from the entire sequence, with positions deemed more relevant contributing more strongly to the output. The weighted sum is a differentiable operation, which is essential for gradient-based training, and it allows the model to smoothly interpolate between attending to one position and spreading attention across many. Each position receives its own unique blend based on its query, so different tokens in the same sequence can produce very different output representations from the same set of Values. Option A is wrong because the element-wise product of Query and Key vectors is not how the output is formed; the Q-K dot product produces the scores, but the output is a weighted combination of Values, not a Q-K product. Option B suggests concatenating the top-scoring Key vectors, but attention uses all positions with varying weights rather than selecting a discrete subset, and it retrieves Values not Keys. Option C describes hard attention, where only the single highest-weighted Value would be selected, but standard Transformer attention is soft, meaning all Values contribute proportionally. This soft blending is one of the key reasons Transformers are so effective, as it lets the model hedge its bets and combine information from multiple relevant positions rather than committing to a single one. The weighted-sum operation is also highly parallelizable, making it efficient on GPUs.