Nybble™
All courses

Guided Course

shiparchitecturecode

Inference Engineering: Inside the Engine

What holds one model server back — arithmetic, memory bandwidth or memory capacity — and every serving technique as a way to move one of them.

0/31
Start course
31 lessons
  1. 1. What Inference Engineering Is6 min
  2. 2. From Text to Tokens to a Forward Pass5 min
  3. 3. The KV Cache and the Two Phases It Creates5 min
  4. 4. The Clocks: TTFT, ITL, TPOT and Throughput5 min
  5. 5. Goodput, Cost per Token and the Workload5 min
  6. 6. Embeddings, Attention and Q, K, V5 min
  7. 7. Why K and V Are Cached and Q Is Not: MQA and GQA5 min
  8. 8. Inside the Accelerator: Compute, Memory and the Bus Between5 min
  9. 9. Compute-Bound or Memory-Bound: The Roofline5 min
  10. 10. The Memory Budget: Weights, Cache and Headroom5 min
  11. 11. Kernels and FlashAttention: The Same Arithmetic, Less Traffic7 min
  12. 12. Quantization: Fewer Bits per Weight5 min
  13. 13. Quantizing and Shrinking the KV Cache5 min
  14. 14. Continuous Batching: Never Leave a Slot Empty5 min
  15. 15. Paged Attention: Memory Handed Out a Block at a Time5 min
  16. 16. Chunked Prefill and the Scheduler's Budgets5 min
  17. 17. Prefix Caching: Never Prefill the Same Tokens Twice5 min
  18. 18. Why Temperature Zero Is Not Deterministic5 min
  19. 19. Speculative Decoding: Guess Ahead, Then Check6 min
  20. 20. The Speedup Equation: How Far Ahead to Guess5 min
  21. 21. Where Drafts Come From, and When Speculation Loses6 min
  22. 22. Constraining and Marking Output8 min
  23. 23. Tensor Parallelism: One Model Across Several GPUs6 min
  24. 24. Pipeline, Data and Expert Parallelism5 min
  25. 25. Anatomy of an Engine: One Loop, Every Technique5 min
  26. 26. Datacenter Engines: vLLM, SGLang and TensorRT-LLM5 min
  27. 27. llama.cpp and Local Inference5 min
  28. 28. Load Testing and Tuning One Server6 min
  29. 29. Encoders, Vision Inputs and Speech5 min
  30. 30. Diffusion Models: A Different Loop5 min
  31. 31. Capstone: Tuning the Running Service5 min
  32. Key takeaways4 min
  33. How to get certified3 min
  34. Your certificate

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack