Why One Model Isn't Enough

Frontier models — Claude Opus, GPT-5, Gemini Ultra — price at $15–75 per million input tokens. Small models — Claude Haiku, GPT-4o-mini, Gemini Flash — price at $0.15–1.25 per million. The gap is 50–100x.

Production traffic analysis across SaaS platforms consistently shows that 60–80% of requests are structurally simple: classification into known categories, data extraction from templated documents, format conversion between schemas, FAQ retrieval with light paraphrasing. These tasks don't require frontier reasoning. Sending them to a $75/M-token model is paying for capability you never use.

The compound AI thesis: compose specialized components into a system where each handles exactly one cognitive function. A classifier routes. A generator produces. A verifier checks. No single component does everything; each does one thing with known cost and latency characteristics.

Databricks reported a 327% surge in compound AI system adoption across their customer base through 2025–2026, with the median enterprise deploying 3–5 models in a single inference pipeline (Databricks State of Data + AI, 2026). The shift is economic before it's architectural — teams hit their inference budget ceiling and decompose.

The three-component minimum

Every compound system that routes traffic needs at least three components:

Classifier: reads the incoming request, outputs a routing decision (which generator tier handles this)
Generator: the model that produces the actual user-facing output
Verifier: checks the generator's output against correctness criteria before returning it

Remove any one and the system degrades. Without the classifier, you can't route cost-efficiently. Without the verifier, you can't catch when a cheap model produces garbage. Without the generator, you have no output.

Cost analysis with real numbers

Consider a production system handling 100,000 requests per day. Average request: 500 input tokens, 300 output tokens.

Baseline — send everything to frontier ($15/M input, $75/M output):

Daily input cost: 100,000 × 500 / 1,000,000 × $15 = $750
Daily output cost: 100,000 × 300 / 1,000,000 × $75 = $2,250
Total daily: $3,000
Monthly: $90,000

Routed — 70% to small model ($1/M input, $2/M output), 30% to frontier:

Small model daily: 70,000 × 500/1M × $1 + 70,000 × 300/1M × $2 = $35 + $42 = $77
Frontier daily: 30,000 × 500/1M × $15 + 30,000 × 300/1M × $75 = $225 + $675 = $900
Routing classifier overhead: 100,000 × 200/1M × $0.25 = $5
Total daily: $982
Monthly: $29,460

Cost reduction: 67%. This aligns with the 40–70% range reported in production deployments (Martian Multi-Model Routing, 2026). The classifier costs less than 1% of the total budget but controls where the other 99% goes.

Where the savings come from

The savings aren't linear with routing percentage. The relationship is:

total_cost = (1 - route_fraction) × frontier_cost + route_fraction × small_cost + classifier_cost

At 50% routing: 48% savings. At 70%: 67% savings. At 90%: 85% savings. But routing accuracy matters — every request incorrectly sent to the small model that needed frontier capability is a quality failure your users see.

The break-even point for adding routing infrastructure: if your monthly inference spend exceeds $5,000, a routing layer pays for itself in the first billing cycle even at conservative 50% routing rates.

The composition pattern

import litellm
from pydantic import BaseModel

class RouteDecision(BaseModel):
    tier: int  # 1 = small, 2 = medium, 3 = frontier
    reason: str

MODELS = {
    1: "anthropic/claude-3-5-haiku-20241022",
    2: "anthropic/claude-sonnet-4-20250514",
    3: "anthropic/claude-opus-4-20250514",
}

async def compound_inference(query: str) -> str:
    # Classify
    route = await classify(query)

    # Generate
    response = await litellm.acompletion(
        model=MODELS[route.tier],
        messages=[{"role": "user", "content": query}],
    )
    output = response.choices[0].message.content

    # Verify
    if not await verify(output, query):
        # Escalate to next tier
        response = await litellm.acompletion(
            model=MODELS[min(route.tier + 1, 3)],
            messages=[{"role": "user", "content": query}],
        )
        output = response.choices[0].message.content

    return output

This is the skeleton. Every production compound system is a variation on this loop: classify, generate, verify, optionally escalate.

Why not just use the best model for everything

Three constraints beyond cost push toward composition:

Latency: frontier models take 2–8 seconds for complex reasoning. Small models respond in 200–800ms. For real-time applications (autocomplete, chat, search), the small model's speed is the product requirement, not just a cost optimization.
Rate limits: frontier model APIs enforce strict tokens-per-minute caps. Distributing load across tiers and providers means you're never fully blocked by a single provider's capacity.
Specialization: a fine-tuned small model on your specific classification task outperforms a general frontier model. The compound system lets you slot in purpose-built components where they dominate.

The compound system isn't a cost hack. It's the recognition that intelligence is cheaper when you decompose it — the same insight that made microservices beat monoliths for scaling, applied to inference.