Transformer Architecture — Advanced

Attention complexity and FlashAttention, MHA vs MQA vs GQA, sparse and sliding-window attention, KV-cache math, Pre/Post-Norm and DeepNorm, attention sinks, head redundancy, and training-stability trade-offs.

Hard50 questions

Sample questions

FlashAttention speeds up attention and cuts memory without changing the maths. What is its core idea?

It prunes low-scoring keys so fewer dot products are computed
It replaces softmax with a linear kernel to avoid the quadratic term
It stores the attention matrix in lower precision to halve memory
It tiles the computation and never materializes the n×n matrix in HBM

Why

FlashAttention is IO-aware: it computes attention block-by-block in fast SRAM using an online-softmax, so the n×n matrix is never written to slow HBM — cutting memory from O(n²) to O(n) with exact results. It does not prune keys, linearize softmax, or rely on low precision for its savings.

FlashAttention FlashAttention-2

Why does the KV cache, not raw FLOPs, usually dominate long-context autoregressive decoding cost?

Each step is memory-bandwidth bound: it reads the whole cache per token
Because the softmax must be recomputed over the entire vocabulary each step
Because the feed-forward network grows with the sequence length
Because positional encodings are recomputed for every past token each step

Why

Decoding one token does little compute but must stream the entire per-layer K/V cache from memory, so it is bandwidth-bound and the cache size sets the ceiling. The FFN and softmax costs are fixed per token, and positional encodings aren't recomputed for the whole history each step.

Multi-Query Attention (Shazeer)GQA: Grouped-Query Attention

Multi-Query Attention (MQA) shares a single Key/Value head across all query heads. What is the main trade-off versus full multi-head attention?

It raises training FLOPs sharply in exchange for lower latency
It removes the need for positional encodings but hurts long-range recall
It increases the number of attention heads while keeping memory fixed
It shrinks the KV cache and speeds decoding, at some quality cost

Why

MQA keeps many query heads but one shared K/V, so the KV cache (and the memory traffic that bounds decoding) shrinks dramatically, trading a little quality. It does not raise training FLOPs, has nothing to do with positional encodings, and reduces — not increases — head count on the K/V side.

Multi-Query Attention (Shazeer)GQA: Grouped-Query Attention

Grouped-Query Attention (GQA) sits between MHA and MQA. How is it configured?

Query heads are split into G groups, each group sharing one K/V head
Every query head keeps its own K/V, but they are averaged after attention
One query head is shared while each position keeps its own K/V
K and V are tied to the same matrix within each individual head

Why

GQA partitions the query heads into G groups that each share a K/V head, so G=heads is full MHA and G=1 is MQA — recovering most MHA quality at near-MQA cache size. The other options misdescribe the grouping.

GQA: Grouped-Query Attention Multi-Query Attention (Shazeer)

A model uses sliding-window attention with window w over L layers. How does it still capture dependencies longer than w?

Information propagates across windows up the layer stack, giving range ~L×w
A special global token attends to every position in each layer
The window is doubled at inference time to cover the full context
Positional encodings extend the receptive field beyond the window

Why

Each layer only attends within w, but stacking layers lets information hop window-to-window, so the effective receptive field grows roughly with depth×window (as in Mistral). The window isn't resized at inference, PEs don't extend range, and a global token is a different scheme (Longformer/BigBird).

Mistral 7B (sliding-window attention)Longformer

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack