Key takeaways

The through-line

Attention is a routing problem, not a magic trick. Every mechanism in this course — from scaled dot-product through GQA and FlashAttention — is answering the same two questions: which positions should inform this one, and how do we compute that without running out of memory.

The ideas that carry

• Scaled dot-product attention is a soft dictionary lookup. Queries, keys and values; the scaling by √d_k exists to keep the softmax out of its saturated regime where gradients die.
• Multi-head attention is about subspaces, not capacity. Splitting into heads lets different heads specialise on different relations — syntactic, positional, semantic — at the same cost.
• Causal masking is the entire difference between reading and writing. Self-attention, cross-attention and masked attention are the same operation with different visibility rules.
• Position is injected, never inherent. Attention is permutation-invariant by construction, so order has to be added. RoPE won because it encodes relative position in a way that degrades gracefully when you extrapolate beyond the trained context; ALiBi bakes the bias into the scores.
• The KV cache is the real constraint on serving. Inference is prefill (parallel) then decode (sequential, one token at a time), and the cache is what makes decode tractable. Its memory grows with batch × context × layers × heads — which is why long context is a memory problem long before it is a compute problem.
• MQA, GQA and MLA are all KV-cache compression. Share keys and values across heads (MQA), share them across groups (GQA), or project them into a latent space (MLA). GQA is the current default because it recovers most of MQA's savings with far less quality loss.
• FlashAttention changed nothing mathematically and everything practically. Same output, tiled so the attention matrix is never materialised in HBM. It is an IO-aware rewrite — proof that on modern accelerators, memory movement is the cost that matters.

What you can now do

Read an attention config and predict its memory profile. Choose between MHA, GQA and MQA for a given latency and quality budget. Explain why a model degrades past its trained context length, and which position scheme would degrade differently.

If one thing sticks

Attention's quality ceiling is set by the math; its production ceiling is set by memory bandwidth. Almost every advance since 2022 has been about the second one.