LLMOps — Core

Model serving pipeline, vLLM and PagedAttention, inference optimization, streaming, LLM APIs, routing and fallbacks, caching, input/output guardrails, OWASP LLM Top 10, observability, cost control.

Medium30 questions

Sample questions

In the LLM serving pipeline, what is the difference between the prefill phase and the decode phase?

Prefill processes all input tokens in parallel and is compute-bound, while decode generates one token at a time and is memory-bandwidth-bound
Prefill loads the model weights into GPU memory and is I/O-bound, while decode runs the forward pass and is compute-bound for every generated token
Prefill tokenizes the input string into token IDs using the vocabulary, while decode converts output token IDs back into readable text
Prefill allocates GPU memory for the entire output sequence up front, while decode fills that memory one token at a time during generation

Why

The serving pipeline has two main computational phases. Prefill processes the entire input prompt in a single batched forward pass, running matrix multiplications across all input tokens in parallel, which saturates GPU tensor cores and makes it compute-bound. Decode then generates output tokens one at a time autoregressively, where each step reads the full model weights and the growing KV cache from GPU high-bandwidth memory but performs relatively little arithmetic per byte loaded, making it memory-bandwidth-bound. Loading model weights happens before serving begins, not during prefill. Tokenization and detokenization are separate CPU-bound stages that take less than a millisecond and are never the bottleneck. GPU memory for the KV cache is allocated before prefill, not as a pre-allocated output buffer. Understanding this distinction matters because optimizations target different phases: prefix caching and prompt compression reduce prefill time, while quantization and batching improve decode throughput. On an A100 with Llama 3 8B, prefill runs at roughly 14,000 tokens per second while decode produces about 100 tokens per second per sequence.

vLLM documentation LLM Inference Performance Engineering (Hugging Face)

What does Time to First Token (TTFT) measure, and why is it the primary UX metric for interactive LLM applications?

TTFT measures the total time to generate the complete response, and it matters because users evaluate quality based on how quickly the full answer appears on screen
TTFT measures the latency of the tokenizer converting the input string to token IDs, and it matters because slow tokenization blocks the entire serving pipeline from starting, which is the standard recommendation for production deployments operating at scale
TTFT measures the time the model spends loading weights into GPU memory before inference begins, which determines whether the system can handle concurrent interactive users
TTFT measures the time from request arrival to the first output token, and it matters because users perceive delays beyond 500ms as sluggish and may abandon the interaction entirely

Why

Time to First Token is the elapsed time from when the server receives a request to when the first output token is emitted to the client. For interactive applications like chatbots and coding assistants, this is the metric users feel most directly because it determines the perceived responsiveness of the system. Research on user experience establishes that responses under 100ms feel instant, 100-500ms feel responsive, 500ms-2s feel slow, and over 2 seconds feels broken. TTFT is dominated by two factors: queue wait time when the server is loaded and prefill computation time which scales with input length. It does not measure total response time, which includes the entire decode phase. Model weight loading happens at server startup, not per request. Tokenization takes less than a millisecond and is never a bottleneck for TTFT. Optimizing TTFT involves prefix caching to skip redundant prefill computation, shorter prompts to reduce prefill time, overprovisioning GPUs to reduce queue depth, and priority queuing to serve interactive requests before batch workloads.

vLLM documentation Building LLM apps for production (Chip Huyen)

What problem does PagedAttention solve in LLM serving, and how does it work?

It splits the attention computation across multiple GPUs by partitioning the query, key, and value matrices along the head dimension for parallel processing
It eliminates KV cache memory fragmentation by dividing the cache into fixed-size blocks allocated on demand, similar to virtual memory paging in operating systems
It compresses attention weights using low-rank approximation to reduce the computational cost of the attention mechanism during both prefill and decode
It reduces the quadratic memory cost of self-attention by computing attention only between nearby tokens within a fixed sliding window rather than attending to all previous tokens

Why

Before PagedAttention, serving frameworks allocated a contiguous block of GPU memory for each request's maximum possible sequence length. A model with a 4K context window reserved 4K tokens worth of KV cache even for a 50-token prompt, wasting 60-80% of KV cache memory through internal fragmentation. PagedAttention borrows the concept of virtual memory paging from operating systems, dividing the KV cache into fixed-size blocks of typically 16 tokens each. Blocks are allocated on demand as the sequence grows, and a block table maps each sequence's logical token positions to physical block locations. Non-contiguous physical blocks can back a logically contiguous sequence. This reduces waste to under 4%, roughly doubling or tripling the number of concurrent sequences on the same GPU. Low-rank approximation describes techniques like GQA or MQA, not PagedAttention. Splitting across GPUs describes tensor parallelism, a different optimization. Sliding window attention is a separate architectural technique used in models like Mistral. PagedAttention also enables efficient prefix sharing where requests with the same system prompt share KV cache blocks using copy-on-write semantics.

vLLM documentation — PagedAttention Deploying LLMs (Philipp Schmid)

You are serving Llama 3 8B on a single A100-80GB in float16. Model weights occupy 16 GB and you reserve 2 GB for overhead. With KV cache costing 128 KB per token, approximately how many concurrent 4K-context sequences can you serve?

Approximately 25 sequences, because KV cache overhead per sequence is much larger than the per-token cost suggests when accounting for attention head duplication
Approximately 20 sequences, because each sequence also requires a full copy of the model's activation memory in addition to its KV cache allocation
Approximately 400 sequences, because AWQ 4-bit quantization is implied for production serving which reduces model weight memory to 4.5 GB and frees much more VRAM
Approximately 100 sequences, because the remaining 54 GB of VRAM divided by the per-sequence KV cache at 4096 tokens gives about 103 concurrent sequences

Why

The calculation proceeds step by step. The A100 has 80 GB of VRAM. At 90% utilization, 72 GB is usable. Model weights in float16 occupy 16 GB and overhead accounts for about 2 GB, leaving 54 GB for KV cache. Each token requires 128 KB of KV cache, so a 4096-token sequence needs 4096 times 128 KB which equals approximately 512 MB or 0.5 GB. Dividing the available 54 GB by 0.5 GB per sequence gives approximately 103 concurrent sequences. The 25-sequence estimate assumes incorrect overhead calculations. The 400-sequence estimate incorrectly assumes quantization that was not specified in the question. The 20-sequence estimate incorrectly claims per-sequence activation memory is needed, but activations are computed in-place during the forward pass and shared across the batch. This type of GPU memory planning calculation is essential before deploying any model because it determines the maximum concurrency and therefore the throughput of the serving infrastructure.

vLLM documentation Deploying LLMs (Philipp Schmid)

What is the key difference between AWQ and GPTQ weight quantization for LLM serving?

GPTQ quantizes all weights uniformly using layer-wise error minimization, while AWQ identifies salient weight channels connected to high-magnitude activations and protects them during quantization for better quality at the same bit width
AWQ is a training-time quantization method applied during pre-training, while GPTQ is a post-training method that quantizes already-trained models without quality calibration, ensuring consistent and reliable performance across all deployment configurations
GPTQ uses 8-bit integers for all weights while AWQ uses mixed 4-bit and 8-bit precision across different layers to balance quality against compression ratio adaptively
AWQ requires a GPU for the quantization process while GPTQ can quantize models entirely on CPU, making GPTQ more accessible for teams without dedicated quantization hardware

Why

Both AWQ and GPTQ are post-training quantization methods that compress model weights to 4-bit integers for efficient serving. GPTQ minimizes the squared error of each layer's output when replacing float16 weights with 4-bit approximations, treating all weights equally in the optimization. AWQ observes that a small fraction of weights, those connected to high-magnitude activations, have a disproportionate impact on output quality. AWQ identifies these salient channels, typically the top 1% by activation magnitude, and scales them up before quantization to protect their precision, then scales down at inference. This targeted protection gives AWQ lower perplexity degradation than GPTQ at the same 4-bit level: within 0.05-0.15 points of float16 for AWQ versus 0.1-0.3 for GPTQ on typical benchmarks. Both methods are post-training, not training-time techniques. Both primarily use 4-bit precision for all layers, not mixed precision. Both require a GPU for the calibration step. AWQ also tends to produce faster inference because its kernel implementation fuses dequantization with matrix multiplication, avoiding a separate dequantize pass.

vLLM Quantization documentation Quantization overview (Hugging Face)

Free account

Take the full module

These are the first few of 30 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack