vLLM and production model servers
vLLM and production model servers
vLLM (Kwon et al. 2023) serves LLMs using PagedAttention, a memory management technique that borrows virtual memory concepts from operating systems to manage the KV cache. Before vLLM, serving frameworks allocated a contiguous block of GPU memory for each request's maximum possible sequence length — a 4K-context model reserved 4K tokens worth of KV cache even for a 50-token prompt. This internal fragmentation wasted 60–80% of KV cache memory. PagedAttention eliminates it.
PagedAttention
In standard attention serving, each sequence's KV cache is stored as a contiguous tensor in GPU memory. If the maximum sequence length is 2,048 tokens and a request has generated 100 tokens so far, the remaining 1,948 token slots sit allocated but empty. With 100 concurrent requests, the wasted memory adds up to tens of gigabytes.
PagedAttention divides the KV cache into fixed-size blocks (default: 16 tokens per block). Blocks are allocated on demand as the sequence grows, like pages in virtual memory. A block table maps each sequence's logical token positions to physical block locations in GPU memory. Non-contiguous physical blocks can back a logically contiguous sequence.
The result: KV cache memory waste drops to under 4% (less than one block per sequence). On an A100-80GB serving Llama 3 8B, this translates to 2–4x more concurrent sequences — and since throughput scales linearly with batch size in the memory-bandwidth-bound decode phase, PagedAttention roughly doubles or triples serving throughput.
PagedAttention also enables efficient memory sharing. Two requests with the same system prompt share the same KV cache blocks for those prefix tokens — the blocks are copy-on-write, allocated once and referenced by both sequences. For a 2,000-token system prompt shared across 50 requests, this saves 50 * 2000 * 128 KB = 12.5 GB of KV cache.
Continuous batching
Static batching groups requests into a fixed-size batch and processes them together until all sequences in the batch finish generating. If one sequence generates 10 tokens and another generates 500, the 10-token sequence holds its GPU slot idle for 490 decode steps. The GPU is doing useful work for that slot only 2% of the time.
Continuous batching (also called iteration-level batching) inserts new requests and retires completed ones at every decode step. When a sequence finishes generating (hits an EOS token or the max length), its slot is immediately given to a waiting request. The GPU stays fully utilized.
vLLM implements continuous batching natively. The scheduler evaluates the request queue at each decode iteration, admits new requests when KV cache blocks are available, and preempts lower-priority requests when memory pressure is high (evicting their KV cache blocks to make room for higher-priority ones).
Setting up vLLM
from vllm import LLM, SamplingParams
llm = LLM(
model="meta-llama/Meta-Llama-3-8B-Instruct",
dtype="float16",
max_model_len=8192,
gpu_memory_utilization=0.90,
tensor_parallel_size=1,
enable_prefix_caching=True,
)
sampling = SamplingParams(
temperature=0.7,
top_p=0.9,
max_tokens=512,
stop=["<|eot_id|>"],
)
prompts = [
"Explain how TCP congestion control works.",
"Write a Python function to merge two sorted lists.",
"What causes a segmentation fault?",
]
outputs = llm.generate(prompts, sampling)
for output in outputs:
prompt = output.prompt
generated = output.outputs[0].text
print(f"Prompt: {prompt[:60]}...")
print(f"Output: {generated[:200]}...")
print(f"Tokens: {len(output.outputs[0].token_ids)}")
print()The gpu_memory_utilization=0.90 parameter tells vLLM to use 90% of GPU memory — the remaining 10% is headroom for PyTorch CUDA allocator overhead and temporary buffers. The enable_prefix_caching=True flag activates automatic prefix sharing for requests with common prefixes.
Running vLLM as an OpenAI-compatible server
Production deployments run vLLM as a persistent HTTP server that exposes the OpenAI Chat Completions API. Any client that speaks the OpenAI protocol works without modification.
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Meta-Llama-3-8B-Instruct \
--dtype float16 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--tensor-parallel-size 1 \
--enable-prefix-caching \
--port 8000Clients connect with the standard OpenAI SDK:
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="meta-llama/Meta-Llama-3-8B-Instruct",
messages=[
{"role": "system", "content": "You are a concise technical assistant."},
{"role": "user", "content": "What is the CAP theorem?"},
],
temperature=0.7,
max_tokens=256,
stream=True,
)
for chunk in response:
if chunk.choices[0].delta.content:
print(chunk.choices[0].delta.content, end="", flush=True)Tensor parallelism
Models that exceed a single GPU's memory are split across multiple GPUs using tensor parallelism. Each linear layer's weight matrix is partitioned column-wise (or row-wise) across GPUs. Each GPU computes its shard of the matrix multiply, then an all-reduce synchronizes the partial results. The communication overhead is one all-reduce per transformer layer per decode step.
For Llama 3 70B on 8xA100-80GB:
- Model weights: ~140 GB in float16, ~17.5 GB per GPU
- KV cache per GPU: ~55 GB available
- All-reduce latency per layer: ~50 microseconds over NVLink
- All-reduce overhead across 80 layers: ~4ms per decode step
- Decode throughput impact: ~10% slowdown versus ideal scaling
python -m vllm.entrypoints.openai.api_server \
--model meta-llama/Meta-Llama-3-70B-Instruct \
--dtype float16 \
--tensor-parallel-size 8 \
--max-model-len 8192 \
--gpu-memory-utilization 0.90 \
--port 8000vLLM handles weight sharding, KV cache partitioning, and all-reduce communication automatically. The tensor-parallel-size must divide the number of attention heads evenly.
Alternative serving frameworks
Text Generation Inference (TGI) from HuggingFace is a Rust+Python server optimized for HuggingFace model hosting. TGI implements continuous batching, FlashAttention, quantization (GPTQ, AWQ, EETQ, bitsandbytes), and speculative decoding. It uses a custom Rust HTTP server (built on axum) that handles request routing, health checks, and streaming. TGI is the backend for HuggingFace's Inference Endpoints product.
TGI's main advantage over vLLM is tight HuggingFace ecosystem integration — models deploy from the Hub with zero configuration. Its disadvantage is slower iteration on new features (the Rust server layer adds development friction) and lower throughput than vLLM on some workloads (Kwon et al. 2023 report 2–4x throughput advantage for vLLM on ShareGPT traces).
docker run --gpus all -p 8080:80 \
ghcr.io/huggingface/text-generation-inference:latest \
--model-id meta-llama/Meta-Llama-3-8B-Instruct \
--max-input-tokens 4096 \
--max-total-tokens 8192 \
--quantize awqTensorRT-LLM from NVIDIA compiles transformer models into optimized TensorRT engines with fused kernels, in-flight batching (NVIDIA's term for continuous batching), and FP8 quantization on Hopper GPUs (H100/H200). TensorRT-LLM delivers the highest absolute throughput on NVIDIA hardware — 20–40% faster than vLLM on H100 for large-batch workloads — but requires a compilation step (model → TensorRT engine) that takes 10–60 minutes and produces hardware-specific binaries. The development cycle is slower and the model coverage is narrower.
SGLang (Zheng et al. 2024) is a serving framework optimized for structured generation: constrained decoding where the output must conform to a JSON schema, regex pattern, or context-free grammar. SGLang compiles the structural constraint into a finite state machine that masks invalid tokens at each decode step, achieving constrained generation with near-zero overhead compared to unconstrained generation. For workloads that require guaranteed JSON output or function-calling format compliance, SGLang outperforms vLLM's guided decoding by 2–5x (measured in tokens/sec during constrained generation).
Choosing a server
For most production workloads: vLLM. It has the broadest model support, the most active development community, strong throughput, and an OpenAI-compatible API. Use TensorRT-LLM when you need maximum throughput on NVIDIA hardware and can tolerate the compilation step. Use SGLang when structured output is the primary workload. Use TGI when deploying via HuggingFace Inference Endpoints.
Benchmarking under load
Throughput and latency numbers from a single request are misleading. Production servers handle concurrent requests, and performance degrades as load increases. Benchmarking requires sending realistic request patterns at varying concurrency levels.
import asyncio
import time
import json
from openai import AsyncOpenAI
client = AsyncOpenAI(base_url="http://localhost:8000/v1", api_key="unused")
PROMPTS = [
"Explain how a B-tree index works in PostgreSQL.",
"Write a Python decorator that retries a function 3 times on exception.",
"What is the difference between a process and a thread?",
"Describe how TLS 1.3 handshake works step by step.",
"Explain consistent hashing and its use in distributed systems.",
]
async def send_request(prompt: str, request_id: int) -> dict:
t0 = time.perf_counter()
ttft = None
output_tokens = 0
stream = await client.chat.completions.create(
model="meta-llama/Meta-Llama-3-8B-Instruct",
messages=[{"role": "user", "content": prompt}],
max_tokens=256,
temperature=0.7,
stream=True,
)
async for chunk in stream:
if chunk.choices[0].delta.content:
if ttft is None:
ttft = time.perf_counter() - t0
output_tokens += 1
total = time.perf_counter() - t0
tpot = (total - ttft) / max(output_tokens - 1, 1) if ttft else 0
return {
"request_id": request_id,
"ttft_ms": ttft * 1000 if ttft else None,
"tpot_ms": tpot * 1000,
"total_ms": total * 1000,
"output_tokens": output_tokens,
}
async def benchmark(concurrency: int, num_requests: int = 50):
semaphore = asyncio.Semaphore(concurrency)
async def bounded_request(prompt, rid):
async with semaphore:
return await send_request(prompt, rid)
tasks = [
bounded_request(PROMPTS[i % len(PROMPTS)], i)
for i in range(num_requests)
]
t0 = time.perf_counter()
results = await asyncio.gather(*tasks)
wall_time = time.perf_counter() - t0
ttfts = [r["ttft_ms"] for r in results if r["ttft_ms"]]
tpots = [r["tpot_ms"] for r in results]
total_tokens = sum(r["output_tokens"] for r in results)
ttfts.sort()
tpots.sort()
print(f"Concurrency: {concurrency}")
print(f" TTFT p50: {ttfts[len(ttfts)//2]:.0f} ms")
print(f" TTFT p95: {ttfts[int(len(ttfts)*0.95)]:.0f} ms")
print(f" TPOT p50: {tpots[len(tpots)//2]:.1f} ms")
print(f" Throughput: {total_tokens / wall_time:.0f} tokens/sec")
print()
async def main():
for concurrency in [1, 4, 16, 32, 64]:
await benchmark(concurrency)
asyncio.run(main())Typical results for Llama 3 8B on A100 (vLLM):
Concurrency: 1
TTFT p50: 45 ms
TTFT p95: 62 ms
TPOT p50: 10.2 ms
Throughput: 95 tokens/sec
Concurrency: 16
TTFT p50: 85 ms
TTFT p95: 210 ms
TPOT p50: 12.8 ms
Throughput: 1520 tokens/sec
Concurrency: 64
TTFT p50: 320 ms
TTFT p95: 890 ms
TPOT p50: 18.5 ms
Throughput: 2850 tokens/secThroughput scales roughly linearly up to concurrency ~32, then sublinearly as KV cache pressure forces request queuing. TTFT degrades more sharply than TPOT because queued requests wait for KV cache blocks before prefill can begin. The inflection point — where TTFT p95 exceeds your SLA — determines your maximum concurrency and, consequently, the number of GPUs you need.