Key takeaways
What this course actually claimed
Every technique moves one of three limits. A model server is held back by the arithmetic a pass must do, the bytes it must move across the memory bus, or the bytes that must fit in memory. Before choosing a technique, find out which of the three is binding. The capstone made the same change twice: made first it was worth nothing, and made third it was worth 38%.
Reading is cheap and writing is not. The running service reads an 8,192-token prompt in 0.24 seconds and takes 1.73 seconds to write 256 tokens back. The prompt is one pass over many positions at once. The reply is one pass for every token.
A decode pass waits on memory, not on arithmetic. One request uses 0.34% of the arithmetic an H100 can do. Nearly all of a pass is the model's weights crossing the memory bus. That is why sharing a pass among many requests costs so little, and why the cache each request brings is what eventually slows it down.
Sharing a pass is close to free until the caches add up. A pass takes 6.80 ms with one conversation in it, 7.89 ms with four and 24.6 ms with fifty. Continuous batching, paged attention, chunked prefill and prefix caching are four answers to one question: what goes into the next pass?
Only the start of a prompt can be cached, so the order of a prompt is a performance setting. Put what never changes first. With three quarters of each prompt shared, prefill takes 65 ms where it took 237. One date at the top of the prompt undoes all of it.
Throughput cannot tell a healthy server from a drowning one. Three numbers get called this service's capacity. At 0.83 requests a second the promise is kept for 99 requests in 100. At 1.68 the most requests finish in time, and 1 in 5 is late. At 2.26 the server finishes the most it can, and none in time. State the percentile with the promise, and test with requests that arrive on their own schedule, not when the last reply comes back.
Temperature zero is not deterministic. One prompt sent 1,000 times gave 80 different completions, because the order in which a kernel adds numbers depends on what else shares the pass. Kernels that fix the order fix the output, at a cost of about 38% of throughput.
Speculation buys speed with arithmetic. At 80% acceptance, four guesses a pass make a stream 2.05 times faster for 1.68 times the arithmetic. A bigger draft is not a better one: in the original paper acceptance rose from 75% to 82% while the speedup fell from 3.4 times to 1.7. On a busy server there is no spare arithmetic to spend, and one team measured speculation running 1.4 to 1.8 times slower.
A second GPU is three different things. Cut every layer across two cards and each token is faster: the weights cross in 2.24 ms where they took 4.48, at the price of 64 exchanges a pass. Cut the stack in two, or run two whole copies, and no token is faster. Two copies also hold fewer conversations, 100 against 115, because each keeps its own weights.
Order matters more than technique. Three changes took the running service from 0.83 requests a second to 6.68 on the same card, and a million output tokens from $3.29 to $0.41: lay the prompt out for the cache, retune the token budget, then put the cache at 8 bits. Those figures are simulated. The method is not: measure the operating point, find what breaks first, change the one thing that moves that limit, and measure again.