Transformer Architecture — Core

Self-attention, Q/K/V, multi-head attention, positional encoding, residuals, layer norm, the FFN, embeddings, and the encoder/decoder stacks.

Medium50 questions

Sample questions

What is the core job of the self-attention mechanism in a Transformer?

It compresses the whole sequence into one fixed-length context vector before the feed-forward layers
It lets each token gather information from other tokens, weighted by learned relevance
It applies a fixed convolution kernel across each token and its immediate neighbours
It carries a recurrent hidden state forward from the first token to the last

Why

Self-attention lets every position attend to every other and form a context-weighted mixture, so meaning depends on the whole sequence. It is not a single bottleneck vector (that was seq2seq), not a fixed convolution (weights are content-dependent), and not recurrent (positions run in parallel).

Attention Is All You Need The Illustrated Transformer

In attention, what roles do the Query, Key, and Value vectors play?

The Query holds the token embedding while the Keys and Values hold the positional encodings
The Query and Keys store the inputs while the Values store the target output labels
All three are identical projections of the token, duplicated for numerical redundancy
The Query says what a token wants, Keys what each offers, Values hold the content

Why

A Query is matched against all Keys to score relevance, and those scores weight the Values that are summed into the output. They are three separate learned projections — not labels, not positional encodings, and generally not identical copies.

Attention Is All You Need (§3.2)The Illustrated Transformer

Why are attention scores divided by the square root of the key dimension before the softmax?

Because it rescales the Value vectors so they all have unit length
Because it is the step that makes the attention weights sum to one
Because large dot products push the softmax into a saturated, low-gradient regime
Because it lowers the parameter count of the projection matrices

Why

A dot product over d_k dimensions has variance ~d_k, so scores grow with dimension and saturate softmax (vanishing gradients); dividing by √d_k restores unit scale. Softmax itself makes weights sum to one, and the scaling touches neither the Values nor the parameter count.

Attention Is All You Need (§3.2.1)D2L: Attention Scoring Functions

For a single query, over which axis is the softmax applied when producing attention weights?

Across all keys, so the per-query weights form a probability distribution
Across the feature dimension within one key vector
Across the batch, so weights sum to one over the examples
Across the value dimension after the weighted sum is taken

Why

Softmax normalizes a query's scores over the set of keys, yielding attention weights that sum to one across positions. Normalizing over features, batch, or values would not give a per-query distribution over the sequence.

The Illustrated Transformer Attention Is All You Need (§3.2.1)

Once the attention weights are known, how is the output for a position produced?

By taking the element-wise product of that position's Query and Key vectors
By concatenating the Key vectors of the top-scoring positions
By taking the attention-weighted sum over all the Value vectors
By selecting the single Value vector with the highest weight

Why

The weights multiply the Values and are summed, so the output is a soft blend of all Values. It is not a hard max-selection, not a Q·K product (that produced the scores), and not a concatenation of keys.

Attention Is All You Need (§3.2)The Illustrated Transformer

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack