Architecture decisions for production

A production ML system processes a request through a specific sequence of decisions: model selection, quantization, serving framework, memory budget, concurrency limits, monitoring, and fallback strategy. Each decision constrains the next. The goal is to find the configuration that meets your quality, latency, and cost requirements simultaneously — and to know when no configuration exists (the requirements are contradictory).

The decision framework

Every production deployment begins with five questions:

What is the task? Generation (chatbot, summarization), classification (sentiment, intent routing), extraction (structured data from documents), code (generation, review, completion), or tool use (function calling, agent loops). The task determines the minimum model capability.

What is the quality bar? Human-level accuracy, good-enough (>90% user satisfaction), or minimum viable (better than regex but doesn't need to be perfect). Higher quality bars restrict you to larger, more expensive models.

What is the latency budget? Real-time (<500ms TTFT, for chat interfaces), interactive (<2s TTFT, for document processing with a loading state), or batch (minutes to hours, for offline pipelines). Latency budgets determine whether you can use frontier API models (200-500ms TTFT) or need self-hosted smaller models (50-100ms TTFT).

What is the cost budget? Per-request cost (determines model choice), monthly cost (determines infrastructure), and how it scales with users (per-token pricing scales linearly; self-hosted scales in GPU-hour steps).

What are the data constraints? Sensitive/regulated data requires self-hosting or BAA-compliant API providers. Public data can go anywhere.

Worked example: code review tool

Requirements: Process pull requests averaging 50K tokens (code diff + context). Quality must catch real bugs — false negatives are expensive. Latency is relaxed (developers wait 5-10 seconds for a review). Cost budget: $500/month for 3,000 reviews/month.

Analysis:

Input size (50K tokens) requires 128K+ context window. This eliminates older models with 4K-32K limits. Remaining candidates: GPT-4o, Claude Sonnet 4, Gemini 2.5 Pro, Llama 3.1 70B.

Quality for code: Claude Sonnet 4 leads on SWE-bench Verified (72.7%) and is the de facto choice for code-centric tasks. GPT-4o is close.

Cost per review with Claude Sonnet 4: (50,000 / 1M) * $3.00 + (2,000 / 1M) * $15.00 = $0.15 + $0.03 = $0.18 per review. At 3,000 reviews/month: $540/month — slightly over budget.

Cost with GPT-4o: (50,000 / 1M) * $2.50 + (2,000 / 1M) * $10.00 = $0.125 + $0.02 = $0.145 per review. At 3,000 reviews/month: $435/month — within budget.

Decision: GPT-4o via API. Falls within cost budget, meets quality bar, and the 300-500ms TTFT + 70 TPS streaming is well within the 5-10s latency budget. No infrastructure to manage.

Worked example: real-time customer chat for a bank

Requirements: TTFT < 500ms (customers expect instant responses). Moderate quality (FAQ-level, not complex reasoning). Strict data privacy — no customer data leaves the bank's infrastructure. High throughput: 1,000 concurrent sessions at peak.

Analysis:

Data privacy eliminates all API providers. Must self-host. The latency requirement (<500ms TTFT) with high concurrency eliminates large models — a 70B model on 4×A100 prefills at ~100-200ms per request, but at 1,000 concurrent sessions, queuing delays would push TTFT well beyond 500ms.

Model choice: Llama 3.1 8B, quantized to INT4 (GPTQ or AWQ). INT4 reduces the 8B-parameter model from 16GB (FP16) to ~4.5GB in memory. At 4K context per session, the KV cache per session is $2 \cdot 8 (\text{KV} \text{heads}, \text{GQA}) \cdot 128 (d_{\text{head}}) \cdot 4096 (\text{context}) \cdot 2 (\text{bytes}) \cdot 32 (\text{layers}) = 512 \text{MB}$ in FP16, or ~128 MB with FP8 KV cache quantization.

Hardware: 4×A100-80GB. With INT4 weights (~4.5GB) plus KV cache overhead, each GPU can serve ~40 concurrent sessions (allocating ~1.5 GB KV cache budget per session with headroom). Total: 160 concurrent sessions per 4-GPU node. For 1,000 concurrent: 7 nodes (28×A100).

Latency: INT4 Llama 3.1 8B prefills 4K tokens in ~15ms on A100 (the model is small and fully compute-bound at short sequences). Decode: ~100+ TPS. TTFT well under 100ms per request.

Cost: 28×A100 on cloud (AWS p4d.24xlarge has 8×A100, so 4 instances): ~$120/hour → $86,400/month. Amortized per conversation (assuming 100K conversations/day): ~$0.03 per conversation. Compare to API pricing for the same volume: 100K conversations × 4K tokens each × $0.15/1M = $60/day for GPT-4o-mini — cheaper on API at this scale, but the bank cannot use external APIs.

Decision: Self-hosted Llama 3.1 8B INT4 on 4 × p4d.24xlarge instances, served via vLLM with continuous batching.

Worked example: document processing pipeline

Requirements: Extract structured data (vendor, amount, date, line items) from 10,000 invoices per day. Batch workload — no latency constraint. Quality must be high (financial data, errors cost money). Cost-sensitive — this is a high-volume, low-margin operation.

Analysis:

Batch workload means latency is irrelevant — maximize throughput per dollar. Each invoice is ~2-4K tokens (image converted to text via OCR, or directly via a vision model). Output is structured JSON, ~200-500 tokens.

A fine-tuned small model (Llama 3.1 8B or even 3B) with JSON-mode structured output can achieve >98% field extraction accuracy after training on 5,000 labeled invoice examples. Fine-tuning cost: ~$50-100 on a single A100 for 3 epochs.

Throughput: Llama 3.1 8B on a single A100 with vLLM continuous batching processes ~200 requests/sec at 3K input + 300 output tokens (with batching, the GPU is fully utilized). 10,000 invoices / 200 per second = 50 seconds of compute.

Cost: A single A100 instance (~$3/hour on AWS spot). Processing 10K invoices takes <1 minute of GPU time. Daily cost: ~$0.05. Even accounting for 24/7 instance availability for on-demand processing: ~$2/hour → $1,440/month.

Compare to API: 10K invoices × 3K tokens input × $0.15/1M (GPT-4o-mini) + 300 tokens output × $0.60/1M = $0.045 + $0.0018 = $0.047 per invoice. Daily: $470. Monthly: $14,100. The fine-tuned self-hosted model is 10x cheaper.

Decision: Fine-tuned Llama 3.1 8B with structured output mode, served on a single A100 via vLLM. Batch all 10K invoices, process in under a minute.

The production checklist

from dataclasses import dataclass
from typing import Optional

@dataclass
class ProductionConfig:
    # Model selection
    model_name: str
    model_size_params: float  # in billions
    quantization: str  # "fp16", "int8", "int4", "fp8"
    
    # Serving
    framework: str  # "vllm", "tgi", "tensorrt-llm", "triton"
    max_context_length: int
    max_batch_size: int
    
    # Hardware
    gpu_type: str  # "a100-80gb", "h100-80gb", "l40s", "a10g"
    num_gpus: int
    tensor_parallel: int
    
    # Memory budget
    model_memory_gb: float
    kv_cache_budget_gb: float
    max_concurrent_requests: int
    
    # Monitoring
    latency_p50_target_ms: float
    latency_p95_target_ms: float
    throughput_target_tps: float
    error_rate_threshold: float
    
    # Fallback
    fallback_model: Optional[str]
    cost_alert_monthly: float

def estimate_config(model_size_b, quantization, context_len, concurrent,
                    gpu_type="a100-80gb"):
    """Estimate serving configuration from requirements."""
    
    gpu_memory = {"a100-80gb": 80, "h100-80gb": 80, "l40s": 48, "a10g": 24}[gpu_type]
    
    # Model memory
    bytes_per_param = {"fp16": 2, "int8": 1, "int4": 0.5, "fp8": 1}[quantization]
    model_mem_gb = (model_size_b * 1e9 * bytes_per_param) / (1024**3)
    
    # KV cache per request (assuming GQA with 8 KV heads, d_head=128)
    n_layers = {8: 32, 70: 80, 405: 126}.get(int(model_size_b), 64)
    kv_per_token_bytes = 2 * 8 * 128 * n_layers * 2  # K+V, heads, d_head, layers, fp16
    kv_per_request_gb = (kv_per_token_bytes * context_len) / (1024**3)
    total_kv_gb = kv_per_request_gb * concurrent
    
    # Total memory needed
    total_mem_needed = model_mem_gb + total_kv_gb + 2  # 2 GB overhead
    num_gpus_needed = max(1, int(total_mem_needed / (gpu_memory * 0.85)) + 1)
    
    return {
        "model_memory_gb": round(model_mem_gb, 1),
        "kv_cache_per_request_gb": round(kv_per_request_gb, 3),
        "total_kv_cache_gb": round(total_kv_gb, 1),
        "total_memory_gb": round(total_mem_needed, 1),
        "gpus_needed": num_gpus_needed,
        "gpu_type": gpu_type,
    }

# Example: Llama 3.1 70B INT4, 4K context, 32 concurrent requests
config = estimate_config(
    model_size_b=70,
    quantization="int4",
    context_len=4096,
    concurrent=32,
    gpu_type="a100-80gb"
)
print("Llama 3.1 70B INT4, 4K context, 32 concurrent:")
for k, v in config.items():
    print(f"  {k}: {v}")

# Example: Llama 3.1 8B FP16, 128K context, 4 concurrent requests
config_8b = estimate_config(
    model_size_b=8,
    quantization="fp16",
    context_len=131072,
    concurrent=4,
    gpu_type="a100-80gb"
)
print("\nLlama 3.1 8B FP16, 128K context, 4 concurrent:")
for k, v in config_8b.items():
    print(f"  {k}: {v}")

Serving frameworks

vLLM — the default for most deployments. PagedAttention for efficient KV cache management (no memory fragmentation), continuous batching, speculative decoding support, quantization (AWQ, GPTQ, FP8). Supports Llama, Mistral, Qwen, and most HuggingFace models. Open-source (Apache 2.0). Typical deployment: python -m vllm.entrypoints.openai.api_server --model meta-llama/Llama-3.1-70B --tensor-parallel-size 4 --quantization awq.

TensorRT-LLM (NVIDIA) — highest raw performance on NVIDIA GPUs through aggressive kernel fusion and FP8 quantization on H100. More complex setup than vLLM but 20-40% higher throughput on supported models. Requires NVIDIA hardware.

Text Generation Inference (TGI, HuggingFace) — Docker-based deployment, good for HuggingFace model hub integration. Simpler than TensorRT-LLM, slightly lower throughput than vLLM on most benchmarks.

Monitoring in production

The four signals to track:

• Latency — p50 and p95 TTFT (time to first token) and TPS (tokens per second). Set alerts at 2x your target.
• Throughput — requests/second and tokens/second aggregate. Capacity planning: if throughput approaches 80% of measured maximum, scale up.
• Error rate — 5xx responses, timeouts, malformed outputs. Threshold: <0.1% for production.
• Cost — daily and monthly token consumption. Set alerts at 80% of budget.

The fallback pattern: if the primary model's error rate exceeds threshold or latency exceeds 3x target, route traffic to a fallback model (typically one tier down — GPT-4o falls back to GPT-4o-mini, self-hosted 70B falls back to 8B). The fallback should always be pre-warmed and ready to receive traffic within seconds.