Key takeaways

The through-line

Serving an LLM is a throughput and memory problem with a quality constraint bolted on. The course's spine is that the levers worth pulling — batching, caching, quantization, routing — are mostly systems engineering, and the biggest wins are unglamorous.

The ideas that carry

• Continuous batching is the single largest throughput win. vLLM's PagedAttention treats KV cache like virtual memory, which is what makes high concurrency affordable at all.
• Prefill and decode have opposite profiles — compute-bound vs. memory-bandwidth-bound — so they should be measured, and often scheduled, separately.
• Quantization is nearly free quality-wise at 8-bit and needs care at 4-bit. PTQ with good calibration data covers most cases; QAT is for when it doesn't. Pruning and distillation are the other two compression paths and they compose.
• Streaming is a product decision with an architecture attached. Time-to-first-token is what users perceive as speed, and it is governed by prefill, not total throughput.
• Routing and fallbacks make a service survive its dependencies. Cheap model first, escalate on difficulty; fall back on provider failure; never let one upstream take the product down.
• Caching has three tiers — exact, semantic, and prefix/KV — with different hit rates and different correctness risks. Semantic caching is where you can quietly serve a wrong answer.
• Guardrails run on input and output and cost latency on every request, so where you put them is a real trade, not a checkbox.
• The OWASP LLM Top 10 is the shared vocabulary for prompt injection, insecure output handling and data leakage — worth knowing by name when talking to a security team.
• Cost control is capacity planning. Token accounting per feature, budgets, and knowing your tokens-per-second per GPU is what turns a surprise invoice into a forecast.

What you can now do

Stand up vLLM and tune it against a latency SLO. Choose a quantization level with evidence. Build an API with streaming, routing, caching and guardrails. Forecast capacity and cost, and run an incident with a runbook that exists beforehand.

If one thing sticks

Latency, throughput and cost are one number wearing three hats. Optimise them together or you will trade one for another without noticing.