Nybble™
The Transformer Architecture

Lesson 15 of 16

The model landscape and choosing

GPT-4o processes a user query through approximately 200B active parameters (estimated, given MoE routing) across ~120 layers, returning a response in 200-500ms TTFT with output streaming at 50-80 tokens/sec. This is one point in a multi-dimensional space of models that vary across quality, cost, latency, context length, and capabilities. Choosing the right model for a production task requires understanding these dimensions concretely.

Frontier closed-source models (2025-2026)

GPT-4o (OpenAI) — the default all-rounder. Strong on text, code, vision, and tool use. ~200K context window. Pricing: $2.50 per 1M input tokens, $10.00 per 1M output tokens. LMSYS Chatbot Arena Elo ~1280. HumanEval: 90.2%. Latency: ~300ms TTFT, ~70 TPS output.

GPT-4o-mini (OpenAI) — smaller, faster variant. Pricing: $0.15 per 1M input tokens, $0.60 per 1M output tokens — 16x cheaper than GPT-4o. Good for classification, summarization, simple generation. HumanEval: 87.2%. TTFT: ~150ms. Sufficient for 80% of production workloads where maximum quality is not required.

Claude Sonnet 4 (Anthropic) — best-in-class for code generation and long-context tasks. 200K context window. Pricing: $3.00 per 1M input, $15.00 per 1M output. Excels at instruction following, structured output, and multi-step reasoning. SWE-bench Verified: 72.7%. The model used by most AI coding assistants.

Claude Opus 4 (Anthropic) — highest quality for complex, multi-step tasks. Same 200K context. Pricing: $15.00 per 1M input, $75.00 per 1M output. Reserved for tasks where quality justifies 5x the cost of Sonnet — complex research, high-stakes document analysis, agentic workflows.

Gemini 2.5 Pro (Google) — 1M+ token context window (the longest in production). Strong multimodal (text, image, video, audio). Pricing: $1.25 per 1M input (under 200K), $2.50 (over 200K); $10.00 per 1M output. The only viable option when your input exceeds 200K tokens.

o3 (OpenAI) — reasoning-optimized model that "thinks" before answering. Variable cost depending on reasoning effort. Excels on math, science, and complex multi-step problems. AIME 2024: 96.7%. Slower TTFT (1-30s depending on problem complexity) but higher accuracy on hard tasks.

Open-weight models

Llama 3.1 405B (Meta) — the largest open-weight model. Apache 2.0-like license (Meta Community License, commercial use allowed above 700M MAU requires permission). 128K context. Matches GPT-4-level on most benchmarks. Requires 8×A100-80GB minimum for FP16 inference. Self-hosting cost: ~$25-40/hour on cloud GPUs.

Llama 3.1 70B (Meta) — the workhorse open model. Fits on 4×A100 (FP16) or 2×A100 (INT4). 128K context. MMLU: 86.0%. HumanEval: 80.5%. Strong enough for production code generation, summarization, and complex instruction following. Self-hosting cost: ~$10-15/hour.

Llama 3.1 8B (Meta) — fits on a single A100 or even a 24GB consumer GPU (RTX 4090) in INT4. 128K context. MMLU: 73.0%. HumanEval: 72.6%. Excellent for real-time applications where latency matters more than maximum quality. Self-hosting cost: ~$1-2/hour.

Mistral Large (Mistral AI) — 128K context, strong reasoning, competitive with Llama 3.1 70B. European company, EU data sovereignty considerations. Apache 2.0 license. Mistral 7B remains one of the best models in its size class, with sliding window attention for efficient long-context handling.

Qwen2.5 72B (Alibaba) — Apache 2.0. Competitive with Llama 3.1 70B on English benchmarks, stronger on Chinese and multilingual tasks. 128K context. Popular for Asian-market deployments.

DeepSeek-V3 (DeepSeek) — 671B total parameters, ~37B active per token (MoE architecture with Multi-head Latent Attention). Extremely cost-efficient inference. Open-weight. Trained for $5.5M in compute — an order of magnitude cheaper than comparable models, demonstrating that algorithmic efficiency can substitute for raw compute budget.

Choosing dimensions

Quality — measured by MMLU (general knowledge), HumanEval/SWE-bench (code), LMSYS Arena Elo (human preference), MATH/AIME (reasoning). No single benchmark captures "quality" — a model that scores highest on MMLU may underperform on code, and vice versa.

Cost — $/1M tokens (API) or $/hour (self-hosted). A 10x price difference between GPT-4o-mini ($0.15/$0.60) and GPT-4o ($2.50/$10.00) means model routing can save 60-80% of inference costs by sending simple queries to the cheap model.

Latency — TTFT (time to first token, perceived responsiveness) and TPS (tokens per second, streaming speed). Real-time applications need TTFT < 500ms. Batch processing doesn't care about latency.

Context window — 4K (legacy), 32K (standard), 128K (current generation), 1M+ (Gemini). Your input size determines the minimum context window. Processing a 50-page PDF (~75K tokens) eliminates any model with <128K context.

Capabilities — tool use (function calling), vision (image input), structured output (JSON mode), code execution. Not all models support all capabilities. Claude and GPT-4o have strong tool use; Llama 3.1 supports it but with lower reliability.

Benchmarks and what they measure

No single number captures model quality. The major benchmarks test different capabilities:

MMLU (Hendrycks et al. 2021) — 57 subjects from elementary math to professional law. 14,042 multiple-choice questions. Tests breadth of knowledge. GPT-4o: ~88%, Llama 3.1 70B: 86%, Llama 3.1 8B: 73%. Saturating — frontier models all score 85-90%, making it less useful for differentiating them.

HumanEval (Chen et al. 2021) — 164 Python programming problems with unit tests. Measures functional correctness of generated code via pass@1 (single attempt) and pass@10 (best of 10). Claude Sonnet 4: 92.0%, GPT-4o: 90.2%, Llama 3.1 70B: 80.5%. SWE-bench Verified is the harder variant — real GitHub issues that require multi-file edits in actual repositories.

LMSYS Chatbot Arena — crowdsourced human preference rankings. Users submit prompts, receive anonymous responses from two models, and vote on which is better. Produces Elo ratings (like chess). As of mid-2025: GPT-4o and Claude Sonnet 4 cluster around 1270-1290 Elo. The most ecologically valid benchmark — it measures what real users actually prefer — but noisy and gameable (models that are verbose or confident tend to win preference votes regardless of accuracy).

MATH (Hendrycks et al. 2021) — competition mathematics. Tests symbolic reasoning and multi-step problem solving. o3: 96.7% on AIME 2024. GPT-4o: ~76%. Open models lag significantly on hard mathematical reasoning.

MT-Bench (Zheng et al. 2023) — multi-turn conversation quality, scored by GPT-4 as judge. Tests instruction following across 8 categories (writing, roleplay, reasoning, math, coding, extraction, STEM, humanities). Scores 1-10. GPT-4o: ~9.2, Llama 3.1 70B: ~8.6, Llama 3.1 8B: ~7.8.

The gap between benchmarks and production performance is real. A model scoring 90% on HumanEval might fail on your specific codebase because it uses an obscure framework or internal DSL. Always evaluate on your own task distribution before committing to a model — synthetic benchmarks are necessary but not sufficient.

The model routing pattern

python
from dataclasses import dataclass

@dataclass
class ModelConfig:
    name: str
    cost_per_1m_input: float
    cost_per_1m_output: float
    max_context: int
    quality_tier: str  # "frontier", "strong", "good", "basic"

MODELS = {
    "gpt-4o": ModelConfig("gpt-4o", 2.50, 10.00, 200_000, "frontier"),
    "claude-sonnet-4": ModelConfig("claude-sonnet-4", 3.00, 15.00, 200_000, "frontier"),
    "gpt-4o-mini": ModelConfig("gpt-4o-mini", 0.15, 0.60, 128_000, "good"),
    "llama-3.1-70b": ModelConfig("llama-3.1-70b", 0.50, 0.70, 128_000, "strong"),
    "llama-3.1-8b": ModelConfig("llama-3.1-8b", 0.10, 0.15, 128_000, "basic"),
}

def route_request(task_type, input_tokens, quality_needed, max_cost_per_request):
    """Select the cheapest model that meets the requirements."""
    candidates = []
    for name, config in MODELS.items():
        if config.max_context < input_tokens:
            continue
        if quality_needed == "frontier" and config.quality_tier != "frontier":
            continue
        if quality_needed == "strong" and config.quality_tier in ("good", "basic"):
            continue
        
        estimated_output = 500  # assume 500 output tokens
        cost = (input_tokens / 1_000_000) * config.cost_per_1m_input + \
               (estimated_output / 1_000_000) * config.cost_per_1m_output
        
        if cost <= max_cost_per_request:
            candidates.append((cost, name, config))
    
    if not candidates:
        return None
    
    candidates.sort(key=lambda x: x[0])
    return candidates[0][1]

# Simple classification task — route to cheapest model
print(route_request("classification", 200, "basic", 0.01))
# -> "llama-3.1-8b"

# Complex code review — needs frontier quality
print(route_request("code_review", 50_000, "frontier", 1.00))
# -> "gpt-4o" (cheaper than claude-sonnet-4 at this input length)

# Long document — needs 200K+ context
print(route_request("summarization", 180_000, "strong", 2.00))
# -> "claude-sonnet-4" or "gpt-4o"

Production implementations use a learned classifier (fine-tuned small model or even regex rules) to categorize incoming requests and route them. Martian, Unify, and OpenRouter provide model routing as a service. Companies like Notion and Intercom route internally — simple autocomplete goes to 8B models, complex analysis goes to frontier models.

When to self-host open models

  • Data privacy — regulated industries (banking, healthcare) where data cannot leave your infrastructure. Self-hosted Llama 3.1 70B keeps all data on-premise.
  • High volume — when per-token API pricing becomes prohibitive. At 100M tokens/day, GPT-4o costs ~$1,250/day ($37,500/month). Self-hosting Llama 3.1 70B on 4×A100 costs ~$360/day ($10,800/month), and the cost is fixed regardless of token volume.
  • Fine-tuning — when you need to train on proprietary data. API providers offer limited fine-tuning; self-hosted models offer full control.
  • Latency — co-locating the model with your application eliminates network round-trip (typically 50-200ms to API providers).

When to use API providers

  • Maximum quality — frontier models (GPT-4o, Claude Sonnet 4, o3) are only available via API. No open model matches them on the hardest tasks.
  • Zero infrastructure — no GPU procurement, no model serving, no monitoring. You pay per token and the provider handles everything.
  • Low volume — at <1M tokens/day, API costs ($2-10/day) are far cheaper than GPU rental ($10-40/day minimum).
  • Rapid iteration — new model versions (GPT-4o → o3) are available immediately via API. Self-hosted models require re-downloading weights and re-deploying.
← Previous