Nybble™
← Back to modules

Inference Optimization — Advanced

Roofline reasoning, FlashAttention internals, chunked prefill and disaggregation, PagedAttention internals, speculative-decoding acceptance, KV reduction, and tensor-parallel serving.

Hard50 questions

Sample questions

1

On the roofline model, why does single-sequence decode sit far to the left (memory-bound) while prefill sits to the right (compute-bound)?

  • Decode has low arithmetic intensity: many bytes read per few FLOPs
  • Decode uses a slower attention kernel than prefill by design
  • Prefill runs at lower precision, raising its arithmetic intensity
  • Prefill skips the KV cache, so it moves fewer bytes overall

Why

Arithmetic intensity (FLOPs per byte) is what places a kernel on the roofline: batch-1 decode reads all weights to produce one token (few FLOPs, many bytes), landing memory-bound, while prefill reuses those reads across many tokens (high intensity), landing compute-bound. Kernel choice, precision, and cache use are not the cause.

The case for 4-bit precisionFlashAttention
2

Increasing the decode batch size raises arithmetic intensity. What ultimately caps how far this helps?

  • The tokenizer throughput, which limits how fast tokens are formed
  • The number of transformer layers, which grows with batch size
  • The precision of the weights, which must drop as batch grows
  • The point where compute saturates or KV-cache memory is exhausted

Why

Batching amortizes weight reads, pushing decode toward the compute roof; gains stop once you hit that roof or run out of KV-cache memory to admit more sequences. Tokenizer speed, layer count, and weight precision don't set this ceiling.

PagedAttention (vLLM)The case for 4-bit precision
3

FlashAttention computes softmax without ever holding a full row of scores. Which technique makes this possible?

  • Online (streaming) softmax that rescales running sums per tile
  • Approximating the softmax with a temperature-scaled sigmoid
  • Precomputing the row maxima on the CPU before the kernel runs
  • Skipping the normalization and dividing only at the very end

Why

FlashAttention uses the online-softmax trick: as it streams tiles, it tracks a running max and running denominator and rescales the accumulated output, so it never materializes a full score row yet stays numerically exact. It is not an approximation, a CPU precompute, or a skipped normalization.

FlashAttentionFlashAttention-2
4

FlashAttention's training backward pass recomputes the attention matrix instead of storing it. Why is this a net win?

  • Recompute from SRAM beats the HBM traffic of storing it
  • The backward pass needs a different, smaller attention matrix
  • Storing the matrix would change the gradients numerically
  • Recomputation lets the model skip the softmax on the backward pass

Why

Since attention is memory-bound, recomputing the scores on the fly (using cached statistics) costs less than the huge HBM reads/writes that storing and reloading the N x N matrix would incur — trading cheap FLOPs for scarce bandwidth. The matrix is the same, gradients are unchanged, and softmax still runs.

FlashAttentionFlashAttention-2
5

Standard FlashAttention parallelizes over batch and heads, which underutilizes the GPU for long sequences at batch size 1. How does FlashAttention-2 address this?

  • It quantizes the attention scores to INT8 for extra speed
  • It also parallelizes across the sequence-length dimension
  • It moves the softmax onto dedicated tensor cores
  • It caps the sequence length to keep all blocks busy

Why

FlashAttention-2 adds parallelism over the sequence-length (query blocks) dimension, so even a single long sequence keeps many GPU thread blocks busy, along with reduced non-matmul work. It does not quantize scores, use special softmax cores, or cap length.

FlashAttention-2FlashAttention
Free account

Take the full module

These are the first few of 50 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack