Inference Optimization — Core

Prefill vs decode, KV cache, MQA/GQA, continuous batching, PagedAttention/vLLM, FlashAttention, speculative decoding, and serving metrics.

Medium50 questions

Sample questions

Why is autoregressive decoding (generating one token at a time) fundamentally slower than a single forward pass over the same tokens?

Each new token needs its own pass over all previous tokens
The model reloads its weights from disk between every token
Each token requires retraining the final layer once
The softmax has to be recomputed over the full vocabulary twice

Why

Generation is sequential: token t+1 cannot start until token t is produced, so you pay one forward pass per token instead of processing them in parallel. Weights are not reloaded from disk, no retraining happens, and the softmax runs once per step.

Attention Is All You Need Fast Inference via Speculative Decoding

What does the KV cache store during generation, and why?

Past tokens' keys and values, so attention skips recomputing them
The prompt tokens in plain text for logging and later replay
The output logits of every step for beam-search backtracking
The model weights in a faster on-chip format for reuse

Why

Attention at each new step needs the keys and values of all previous tokens; caching them avoids recomputing the whole prefix every step, turning O(n^2) recompute into O(n) per token. It does not store text, logits, or weights.

Attention Is All You Need PagedAttention (vLLM)

LLM inference has two phases with very different performance profiles. How are they best characterized?

Prefill runs in FP32 and decode runs in INT8 by default
Prefill decodes one token; decode processes the whole prompt at once
Both phases are identical in cost, differing only in output length
Prefill is compute-bound over the prompt; decode is memory-bound per token

Why

Prefill processes all prompt tokens in parallel (a big, compute-bound matmul), while decode generates one token at a time and is dominated by reading weights and the KV cache from memory (bandwidth-bound). The other options invert the phases or their precision.

PagedAttention (vLLM)The case for 4-bit precision

You measure a chatbot and see a long wait before the first token, then tokens stream quickly. Which metric describes that initial wait?

Tokens per second, the sustained generation throughput
Total generation length across the whole response
Requests per second the server can sustain concurrently
Time to first token, set by prompt prefill

Why

The gap before the first token is time-to-first-token (TTFT), set mostly by prefilling the prompt. Tokens/sec and per-token latency describe the streaming phase, and requests/sec is a throughput measure, not the first-token wait.

vLLM documentation PagedAttention (vLLM)

Why does the KV cache, rather than the model weights, often limit how many requests you can batch together?

It must be duplicated once per attention head in every request
Its memory grows with sequence length and batch, unlike fixed weights
It is stored in FP32 while the weights sit in INT8
It has to be re-uploaded to the GPU on every decode step

Why

Weights are a fixed cost loaded once, but KV-cache memory scales with (tokens x layers x batch), so long or numerous sequences exhaust memory and cap the batch. It is not FP32-only, not re-uploaded each step, and not duplicated per head beyond its normal shape.

PagedAttention (vLLM)Multi-Query Attention

Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack