Nybble™
Compound AI Systems

Lesson 2 of 13

The Classifier-Generator-Verifier Triad

The Classifier-Generator-Verifier Triad

The minimum viable compound system runs three components in sequence. The classifier reads the input and emits a routing decision. The generator produces the output using the model tier the classifier selected. The verifier checks correctness before returning.

Classifier

A lightweight model — Haiku-class at ~$0.25/M tokens, or a fine-tuned BERT at zero marginal token cost — that performs a single function: read the incoming query and output a structured routing decision.

The classifier's output is small and constrained:

class ClassifierOutput(BaseModel):
    tier: int          # 1, 2, or 3
    confidence: float  # 0.0 to 1.0
    category: str      # "extraction", "reasoning", "creative", "code"

The classifier sees the full input but produces only routing metadata. Its latency budget is tight — 50ms for a fine-tuned classifier, 100–200ms for an LLM-based one. Every millisecond here delays the entire pipeline.

Training data for the classifier comes from production logs: take historical requests, label them with the cheapest model tier that produced acceptable output, and fine-tune. A dataset of 5,000–10,000 labeled examples produces a classifier with 88–94% routing accuracy (Ding et al., "Hybrid LLM," 2024).

Generator

The component that produces the user-facing output. Three tiers map to three cost-capability points:

  • Tier 1 (small, fast, cheap): Claude Haiku, GPT-4o-mini. 200–500ms latency. $0.25–1.25/M tokens. Handles classification, extraction, formatting, simple Q&A.
  • Tier 2 (medium): Claude Sonnet, GPT-4o. 500–2000ms. $3–5/M tokens. Handles multi-step reasoning, code generation, nuanced summarization.
  • Tier 3 (frontier): Claude Opus, GPT-5. 2000–8000ms. $15–75/M tokens. Handles novel reasoning, complex code architecture, ambiguous creative tasks.

The generator is the only component the user "sees." Classifier and verifier are invisible infrastructure. This means the generator's prompt engineering gets the most attention — system prompts, few-shot examples, output format instructions all live here.

GENERATOR_CONFIG = {
    1: {
        "model": "anthropic/claude-3-5-haiku-20241022",
        "max_tokens": 1024,
        "temperature": 0.1,
        "timeout": 5,
    },
    2: {
        "model": "anthropic/claude-sonnet-4-20250514",
        "max_tokens": 4096,
        "temperature": 0.3,
        "timeout": 15,
    },
    3: {
        "model": "anthropic/claude-opus-4-20250514",
        "max_tokens": 8192,
        "temperature": 0.4,
        "timeout": 30,
    },
}

async def generate(query: str, tier: int, system_prompt: str) -> str:
    config = GENERATOR_CONFIG[tier]
    response = await litellm.acompletion(
        model=config["model"],
        messages=[
            {"role": "system", "content": system_prompt},
            {"role": "user", "content": query},
        ],
        max_tokens=config["max_tokens"],
        temperature=config["temperature"],
        timeout=config["timeout"],
    )
    return response.choices[0].message.content

Verifier

Checks the generator's output against correctness criteria. Three verification strategies, ordered by increasing sophistication:

  • Schema validation: does the output match the expected JSON structure, contain required fields, respect length constraints? Zero LLM cost — pure code.
  • Heuristic checks: is the response too short (< 50 tokens for a question that needs explanation)? Does it contain hedging language ("I'm not sure", "it depends") above a threshold? Does it repeat the question back? Fast pattern matching.
  • LLM-as-judge: a second model evaluates the output for factual consistency, completeness, safety. Costs tokens but catches semantic failures that schema validation misses.
async def verify(output: str, query: str, tier: int) -> VerifyResult:
    # Level 1: schema check (free, <1ms)
    if not passes_schema(output):
        return VerifyResult(passed=False, reason="schema_fail")

    # Level 2: heuristics (free, <5ms)
    if len(output.split()) < 20 and expects_explanation(query):
        return VerifyResult(passed=False, reason="too_short")

    if hedging_ratio(output) > 0.3:
        return VerifyResult(passed=False, reason="low_confidence_language")

    # Level 3: LLM judge (costs tokens, ~200ms)
    # Only run for tier 1/2 outputs where errors are more likely
    if tier < 3:
        judge_score = await llm_judge(query, output)
        if judge_score < 0.7:
            return VerifyResult(passed=False, reason="judge_reject")

    return VerifyResult(passed=True, reason="ok")

The verifier's latency budget: 200ms for schema + heuristics, up to 500ms if including an LLM judge. The judge adds cost — typically a Haiku-class call — but catches 15–25% of quality failures that schema-only verification misses.

When to add a fourth component

  • Reranker: when you generate multiple candidates (N completions) and need to select the best one. Adds 200–400ms but improves output quality by 10–20% on subjective tasks.
  • Post-processor: format transformation after generation — stripping markdown, converting to HTML, injecting citations. Pure code, adds < 10ms.
  • Retriever (RAG): fetches context from a vector store or search index before generation. Adds 100–300ms but grounds the output in your data.

When to resist adding components

Each component adds:

  • Latency: minimum 50ms network overhead per component, plus compute time
  • Failure surface: each API call can timeout, rate-limit, or return garbage
  • Debugging complexity: tracing a failure through 5 components is harder than through 3

The latency budget for the full triad: classifier ~50ms + generator ~500–3000ms + verifier ~200ms = total 750–3250ms. Adding a retriever and reranker pushes you to 1200–4000ms. For real-time applications, every added component requires justification against user-perceived latency.

← Previous