LLMOps — Advanced

GPU memory math, KV cache sizing, speculative decoding acceptance rates, continuous batching economics, quantization trade-offs, circuit breaker design, semantic cache tuning, OWASP attack chains, cost calculus, capacity planning.

Hard30 questions

Sample questions

Llama 3 8B in float16 uses 16 GB for weights on an A100-80GB with 90% utilization and 2 GB overhead. KV cache costs 128 KB per token. If you switch to AWQ 4-bit (4.5 GB weights), how many additional concurrent 4K-context sequences can you serve compared to float16?

About 22 additional sequences, because the 11.5 GB freed from weight compression translates to roughly 22 more 4K sequences at 0.5 GB of KV cache each
About 10 additional sequences, because the freed VRAM is mostly consumed by the increased dequantization buffer overhead that AWQ requires during inference
About 90 additional sequences, because AWQ reduces both the model weights and the KV cache memory proportionally, doubling the effective capacity
About 50 additional sequences, because 4-bit quantization halves all memory requirements uniformly including KV cache, activations, and CUDA context overhead

Why

With float16 weights at 16 GB, available KV cache memory is 80 times 0.9 minus 16 minus 2, which equals 54 GB, supporting 54 divided by 0.5 equals 108 concurrent 4K sequences. With AWQ 4-bit weights at 4.5 GB, available memory becomes 80 times 0.9 minus 4.5 minus 2, which equals 65.5 GB, supporting 65.5 divided by 0.5 equals 131 sequences. The difference is 131 minus 108, approximately 22 additional sequences. The 10-sequence estimate incorrectly assumes large dequantization overhead, but AWQ's kernel fuses dequantization with the matrix multiply, adding negligible memory. The 90-sequence estimate incorrectly claims KV cache is also reduced, but KV cache stores attention states in float16 regardless of weight quantization. The 50-sequence estimate incorrectly assumes quantization halves all memory, but only model weights are quantized. This calculation demonstrates why weight quantization is valuable beyond the throughput gains from smaller matrix multiplies: the freed VRAM goes directly to KV cache, enabling more concurrent sequences. The 21% capacity increase from 108 to 131 sequences translates directly to higher throughput since decode throughput scales linearly with batch size.

vLLM documentation Quantization overview (Hugging Face)

In speculative decoding, a draft model proposes K=5 tokens and the target model verifies them in a single forward pass. If the acceptance rate alpha is 0.7, what is the expected number of tokens produced per target forward pass?

Exactly 5 tokens, because the target model processes all 5 draft tokens in parallel and the rejection sampling only affects the probability distribution, not the count
3.5 tokens, computed as K times alpha, which gives the expected number of accepted tokens from the draft sequence before any rejection occurs
Approximately 3.3 tokens, computed as 1/(1-alpha) = 1/0.3, representing the geometric distribution of the first rejection point plus the resampled token
1.0 token, because speculative decoding is lossless and the target model still ultimately generates one token per forward pass regardless of the draft

Why

In speculative decoding, the expected number of tokens produced per target forward pass follows a geometric distribution. Each draft token is accepted with probability alpha, and the first rejection triggers a resample using the target model's distribution. The expected number of accepted tokens before the first rejection is alpha plus alpha-squared plus alpha-cubed and so on, which sums to alpha divided by (1 minus alpha). Adding the one resampled token at the rejection point gives a total expectation of 1 divided by (1 minus alpha). With alpha equals 0.7, this is 1 divided by 0.3, approximately 3.33 tokens per target forward pass. The answer of exactly 5 is wrong because not all draft tokens are accepted, only those matching the target's distribution. The K times alpha calculation of 3.5 confuses the expected number of acceptances with the geometric expectation, which includes the guaranteed resample token. The answer of 1.0 misunderstands the mechanism entirely, as the whole point is that verification produces multiple tokens per forward pass. This 3.3x speedup is free in terms of output quality because the modified rejection sampling scheme guarantees the output distribution is mathematically identical to the target model alone.

vLLM Speculative Decoding Building LLM apps for production (Chip Huyen)

You observe that your vLLM server's KV cache utilization is at 95% and TTFT p95 has spiked from 200ms to 3 seconds. What is happening and what is the correct remediation?

The model weights are being swapped between GPU memory and CPU RAM due to memory pressure, causing the forward pass to stall while waiting for weight transfers
New requests are queuing because there are insufficient free KV cache blocks to allocate for prefill, and the queue wait time dominates TTFT at near-capacity utilization
The GPU tensor cores are thermally throttling due to sustained high utilization, reducing compute throughput and increasing the time for each prefill computation step, which provides a more reliable approach for production deployment environments at scale
The tokenizer is bottlenecking on CPU because the high batch size requires tokenizing many concurrent prompts, and tokenization time scales linearly with concurrency

Why

When KV cache utilization approaches capacity, the vLLM scheduler cannot allocate KV cache blocks for new requests because all available blocks are occupied by in-progress sequences. New requests must wait in a queue until a running sequence completes and releases its blocks. This queue wait time is added to the normal TTFT, which explains the spike from 200ms to 3 seconds. At 95% utilization, only 5% of KV cache blocks are free, meaning only a few new sequences can be admitted at any moment. The correct remediation is to reduce the number of concurrent sequences by either adding GPU capacity, reducing the maximum sequence length, enabling KV cache quantization to FP8 which doubles effective capacity, or implementing request admission control that rejects or queues requests gracefully at lower utilization thresholds. Model weights are not swapped to CPU in standard serving configurations. Thermal throttling would affect all operations uniformly, not selectively spike TTFT. Tokenization takes under 1ms regardless of concurrency and is never the bottleneck. At 100% utilization, requests are rejected outright with a 503 error.

vLLM documentation Deploying LLMs (Philipp Schmid)

Why does continuous batching provide higher GPU throughput than static batching for LLM decode, even though both process the same total tokens?

Continuous batching uses a more efficient attention kernel that reduces the per-token compute cost by fusing the query-key-value projections into a single matrix operation
Continuous batching splits long sequences into shorter chunks processed across multiple decode steps, reducing the per-step KV cache read cost and allowing higher effective batch sizes and this has been validated across diverse production workloads on multiple hardware platforms
Continuous batching pre-sorts requests by output length so that sequences completing at similar times are grouped together, minimizing the variance in generation length per batch
Continuous batching eliminates idle GPU slots by replacing completed sequences immediately, so the memory-bandwidth cost of reading model weights is amortized over more active sequences per step

Why

Decode is memory-bandwidth-bound: each step reads the entire model weights from high-bandwidth memory and reuses them across all sequences in the batch. The throughput gain from batching comes from amortizing this weight-read cost over more sequences. In static batching, when a short sequence finishes, its GPU slot sits idle until the entire batch completes. If one sequence generates 10 tokens and another generates 500, the short sequence's slot wastes compute for 490 steps. Continuous batching fills that idle slot immediately with a new request from the queue, ensuring the model weights read on every step are amortized over the maximum number of active sequences. The total tokens processed may be the same, but continuous batching processes them with fewer idle slot-steps, meaning each weight read serves more useful work. The attention kernel is the same in both approaches. Pre-sorting by output length is impractical because output lengths are unknown in advance. Continuous batching does not chunk sequences; it operates at the iteration level, inserting and retiring whole sequences. The throughput difference is typically 2-4x under realistic workloads where output lengths vary significantly.

vLLM documentation LLM Inference Performance Engineering (Hugging Face)

A serving deployment uses FP8 KV cache quantization. How does this differ from weight quantization, and what is the quality-capacity trade-off?

FP8 KV cache quantization reduces the precision of the attention logits during softmax computation, while weight quantization reduces the precision of the matrix multiply inputs
FP8 KV cache quantization and weight quantization are the same operation applied at different pipeline stages, both compressing the same underlying model tensors with identical quality impact, based on extensive empirical measurements across multiple model families and serving frameworks
FP8 KV cache quantization applies only during prefill to speed up the initial prompt processing, while weight quantization applies only during decode to reduce the per-token generation cost
FP8 KV cache quantization compresses the stored attention key-value states to 8-bit floating point, halving cache memory with minimal quality loss, while weight quantization compresses the model parameters and affects the feedforward computation instead

Why

KV cache quantization and weight quantization target different tensors and introduce error in different parts of the computation. Weight quantization compresses the model's learned parameters, introducing error in every linear transformation and feedforward network computation. KV cache quantization compresses the key and value tensors stored during attention, introducing error in the attention score and weighted-value computations but leaving the feedforward path unaffected. FP8 KV cache halves the cache memory from float16 to 8-bit float, doubling the effective KV cache capacity and therefore doubling the maximum concurrent sequences. The quality impact is typically minimal, with perplexity increase under 0.05 on standard benchmarks, because the attention computation is relatively robust to small quantization errors in stored KV states. INT4 KV cache is more aggressive and can degrade long-context recall. KV cache quantization is not the same as weight quantization since they compress different data and affect different computations. KV cache quantization applies during both prefill and decode, not just prefill. The two techniques are complementary: a deployment can use AWQ 4-bit weights and FP8 KV cache simultaneously for maximum memory efficiency.

vLLM documentation — KV Cache Quantization overview (Hugging Face)

Free account

Take the full module

These are the first few of 30 questions. A free account opens the rest as a scored drill.

Every question in this module
Instant feedback and supporting reading
Your score and progress, saved

Free · your email is used for progress only.

About Nybble^™

The AI space moves fast.
Nybble^™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack

Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack

Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack