The model serving stack

A request arrives as a string. The serving stack transforms that string into output tokens through six sequential stages: tokenization, KV cache allocation, prefill, decode, sampling, and detokenization. Each stage has a distinct compute profile, and the bottleneck shifts between them depending on the prompt length, generation length, batch size, and hardware.

The six-stage pipeline

Tokenization converts the input string into a sequence of integer token IDs using the model's vocabulary. Llama 3 uses a BPE tokenizer with a 128K vocabulary. A 1,000-word English prompt tokenizes to roughly 1,300 tokens. Tokenization is CPU-bound and takes <1ms for typical prompts — it is never the bottleneck.

KV cache allocation reserves GPU memory for the key-value tensors that the attention mechanism will produce during prefill and reuse during decode. For Llama 3 8B with 32 layers, 8 KV heads (GQA), and head dimension 128 in float16:

kv_bytes_per_token = 2 * 32 * 8 * 128 * 2 = 131,072 bytes = 128 KB

A 4,096-token prompt requires 4096 * 128 KB = 512 MB of KV cache. A 128K-token context: 128,000 * 128 KB = 16 GB. This is allocated up front, before any computation begins. If the server cannot allocate the KV cache, the request is queued or rejected.

Prefill (also called the prompt processing phase) runs the full transformer forward pass over the entire input sequence in parallel. Every input token attends to every preceding token in a single batched matrix multiplication. Prefill is compute-bound — it saturates GPU tensor cores. On an A100-80GB with Llama 3 8B in float16, prefill processes roughly 12,000–15,000 tokens/second. A 2,000-token prompt completes prefill in ~150ms.

Decode (autoregressive generation) produces one token at a time. Each decode step runs the full transformer forward pass for a single new token, which attends to all previous tokens via the KV cache. Decode is memory-bandwidth-bound — each step reads the entire model weights and the growing KV cache from HBM but performs relatively little arithmetic per byte loaded. On an A100 with Llama 3 8B, a single decode step takes ~10ms, yielding ~100 tokens/second per sequence.

Sampling selects the next token from the model's output logit distribution. Temperature scaling, top-p (nucleus) sampling, top-k filtering, and repetition penalties all apply here. Sampling is computationally trivial — microseconds per token.

Detokenization converts the output token IDs back to a string. Like tokenization, this is CPU-bound and takes <1ms.

The critical insight: prefill and decode have opposite compute profiles. Prefill is compute-bound (high arithmetic intensity, GPU cores are the bottleneck). Decode is memory-bandwidth-bound (low arithmetic intensity, HBM bandwidth is the bottleneck). A serving stack that optimizes for one without considering the other will underperform.

Timing the pipeline

import time
from vllm import LLM, SamplingParams

llm = LLM(model="meta-llama/Meta-Llama-3-8B-Instruct", dtype="float16")
sampling = SamplingParams(temperature=0.7, top_p=0.9, max_tokens=256)

prompt = "Explain the difference between TCP and UDP in detail."

t0 = time.perf_counter()
outputs = llm.generate([prompt], sampling)
t1 = time.perf_counter()

result = outputs[0]
prompt_tokens = len(result.prompt_token_ids)
output_tokens = len(result.outputs[0].token_ids)
total_time = t1 - t0

print(f"Prompt tokens:  {prompt_tokens}")
print(f"Output tokens:  {output_tokens}")
print(f"Total time:     {total_time*1000:.0f} ms")
print(f"Throughput:     {output_tokens / total_time:.0f} tokens/sec")

On a single A100-80GB, typical results for Llama 3 8B:

Prompt tokens:  12
Output tokens:  256
Total time:     2680 ms
Throughput:     95 tokens/sec

The 2,680ms breaks down roughly as: tokenization <1ms, KV cache allocation ~5ms, prefill ~15ms (short prompt), decode ~2,560ms (256 tokens at ~10ms each), sampling included in decode, detokenization <1ms. Decode dominates — 95% of total time. With a 4,000-token prompt, prefill would take ~300ms but decode would still dominate for any generation longer than ~30 tokens.

Throughput versus latency

These are different metrics that optimize differently.

Latency is what one user experiences:

Time to first token (TTFT): time from request arrival to the first output token. Dominated by prefill. For a 2,000-token prompt on Llama 3 8B / A100: ~150ms. For a 32,000-token prompt: ~2.5 seconds. TTFT determines whether the response feels instant (<500ms) or sluggish.
Time per output token (TPOT): time between consecutive output tokens during decode. For Llama 3 8B on A100 with batch size 1: ~10ms/token (100 tok/s). Users reading streaming text perceive >100ms/token as choppy.
End-to-end latency: TTFT + (output_length TPOT). A 256-token response with 150ms TTFT and 10ms TPOT: `150 + 256 10 = 2,710ms`.

Throughput is what the operator measures: total tokens generated per second across all concurrent requests on a GPU. A single request on Llama 3 8B / A100 generates ~100 tok/s. But with continuous batching at batch size 32, the GPU generates ~2,400 tok/s total — 75 tok/s per request, but 24x total throughput. Batching amortizes the memory-bandwidth cost: the model weights are read from HBM once per decode step and reused across all sequences in the batch.

The tradeoff: higher batch sizes increase throughput but increase per-request latency (each decode step takes longer because it processes more sequences).

The serving triangle: quality, speed, cost

Every deployment makes tradeoffs across three axes:

Quality: model size, quantization level, sampling parameters. Llama 3 70B produces better outputs than 8B but costs more to serve. 4-bit quantization reduces quality slightly but halves memory and increases throughput.
Speed: TTFT and TPOT targets. Interactive chat needs <500ms TTFT and <50ms TPOT. Batch processing (summarizing 10,000 documents) tolerates seconds of latency per request.
Cost: GPU-hours per million tokens generated. An A100 costs ~$2/hour on cloud. Llama 3 8B on one A100 generates ~2,400 tok/s at batch 32 = 8.64M tokens/hour = ~$0.23 per million tokens. Llama 3 70B on 8xA100 generates ~800 tok/s at batch 16 = 2.88M tokens/hour at $16/hour = ~$5.55 per million tokens.

Pick two. A 70B model with low latency requires expensive multi-GPU setups. A cheap deployment with good quality means tolerating higher latency (larger batches, queuing). A fast, cheap deployment means a smaller or quantized model.

Concrete hardware numbers

Llama 3 8B on a single A100-80GB (float16):

Model weights: ~16 GB
Available for KV cache: ~60 GB
Max concurrent sequences (4K context): ~120
Prefill throughput: ~14,000 tokens/sec
Decode throughput (batch 1): ~100 tokens/sec
Decode throughput (batch 32): ~2,400 tokens/sec total
Cost per 1M output tokens (at $2/hr A100): ~$0.23

Llama 3 70B on 8xA100-80GB (float16, tensor parallel):

Model weights: ~140 GB (spread across 8 GPUs, ~17.5 GB each)
Available for KV cache per GPU: ~55 GB
Max concurrent sequences (4K context): ~50
Prefill throughput: ~4,000 tokens/sec
Decode throughput (batch 1): ~30 tokens/sec
Decode throughput (batch 16): ~400 tokens/sec total
Cost per 1M output tokens (at $16/hr for 8xA100): ~$11.10

Llama 3 8B quantized to 4-bit (AWQ) on a single A100:

Model weights: ~4.5 GB
Available for KV cache: ~72 GB
Max concurrent sequences (4K context): ~180
Decode throughput (batch 64): ~4,800 tokens/sec total
Cost per 1M output tokens: ~$0.12

Monitoring the pipeline

Production serving requires tracking each stage independently. vLLM exposes Prometheus metrics that map to the pipeline stages:

from prometheus_client import Histogram, Gauge

TTFT_HISTOGRAM = Histogram(
    "llm_time_to_first_token_seconds",
    "Time from request receipt to first generated token",
    buckets=[0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0],
)

TPOT_HISTOGRAM = Histogram(
    "llm_time_per_output_token_seconds",
    "Time between consecutive output tokens",
    buckets=[0.005, 0.01, 0.02, 0.05, 0.1, 0.25],
)

BATCH_SIZE_GAUGE = Gauge(
    "llm_current_batch_size",
    "Number of sequences currently being processed",
)

KV_CACHE_USAGE = Gauge(
    "llm_kv_cache_usage_ratio",
    "Fraction of KV cache blocks currently allocated",
)

The key operational signals: TTFT p95 (are prompts processing fast enough?), TPOT p95 (is decode keeping up?), KV cache utilization (are we about to start rejecting requests?), and batch size (are we efficiently utilizing the GPU?). When KV cache utilization hits 90%, new requests start queuing. When it hits 100%, requests are rejected with a 503.