Nybble™
← Back to modules

Compound AI Systems — Core

Classifier-generator-verifier triad, system contracts, routing strategies, cascade and fallback patterns, AI gateways, semantic caching, cost optimization, shadow testing, MCP+A2A protocol stack, agent registries, compound system observability.

Medium30 questions

Sample questions

1

What is the compound AI thesis and why does it reduce inference costs?

  • The thesis is that fine-tuning a single model on all tasks produces better results than using multiple models, because a unified model shares learned representations across tasks
  • The thesis is that composing specialized components where each handles one cognitive function, with a classifier routing, a generator producing, and a verifier checking, avoids sending simple tasks to expensive frontier models
  • The thesis is that replacing LLM inference with traditional rule-based systems for the majority of requests eliminates API costs entirely while maintaining acceptable quality on common query types, which is the standard recommendation for production compound AI systems at scale
  • The thesis is that running multiple copies of the same model in parallel and taking the majority vote improves accuracy enough to justify the increased compute cost through better reliability

Why

The compound AI thesis states that production systems should compose specialized components into a system where each handles exactly one cognitive function. A classifier routes requests to the appropriate model tier. A generator produces the output using the selected model. A verifier checks correctness before returning. Since 60-80% of production traffic is structurally simple, these requests can be handled by cheap models at $0.15-1.25 per million tokens instead of frontier models at $15-75. With 70% routing to small models, cost reduction reaches 67%. Databricks reported a 327% surge in compound system adoption in 2025-2026, with the median enterprise deploying 3-5 models in a single inference pipeline. The thesis is not about fine-tuning one model, majority voting, or replacing LLMs with rule-based systems. The shift is economic before architectural: teams hit their inference budget ceiling and decompose. The break-even point is approximately $5,000 monthly inference spend.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
2

What are the three components of the minimum viable compound AI system?

  • Retriever, generator, and ranker forming the standard RAG pipeline that grounds LLM output in external documents retrieved from a vector database
  • Encoder, decoder, and attention mechanism forming the transformer architecture that processes input tokens and generates output tokens through self-attention
  • Tokenizer, model, and post-processor forming the inference pipeline that converts text input to token IDs, processes them through the neural network, and converts output tokens back to text and has been validated across diverse production compound system deployments
  • Classifier that routes requests to model tiers, generator that produces the user-facing output, and verifier that checks correctness before returning, with each component having a defined contract

Why

Every compound system that routes traffic needs at least three components. The classifier reads the incoming request and outputs a routing decision determining which generator tier handles it. The generator is the model that produces the actual user-facing output, selected from tiers mapping to different cost-capability points. The verifier checks the generator's output against correctness criteria including schema validation, heuristic checks, and optionally an LLM-as-judge evaluation before returning. Remove any one and the system degrades: without the classifier you cannot route cost-efficiently, without the verifier you cannot catch when a cheap model produces garbage, and without the generator you have no output. The RAG pipeline, transformer architecture, and inference pipeline are different system decompositions that do not describe the compound AI routing pattern. The classifier costs less than 1% of the total budget but controls where the other 99% goes.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
3

What are the four routing strategies for compound AI systems, ordered from simplest to most sophisticated?

  • Rule-based using keyword matching, classifier-based using a trained model, LLM-based using a small model to evaluate query complexity, and cascade where cheap models try first and escalate on failure
  • Static assignment, round-robin, weighted random, and least-connections, following the standard load balancing strategies used in web server infrastructure, based on empirical data from compound AI systems processing millions of daily requests
  • Synchronous, asynchronous, streaming, and batch, ordered by the timing pattern of how requests flow through the classifier-generator-verifier pipeline
  • Single-model, dual-model, multi-model, and ensemble, ordered by the number of models involved in processing each request through the compound pipeline

Why

Four routing strategies trade off latency overhead against routing accuracy. Rule-based routing uses regex, keywords, and message length with under 1ms latency and zero cost but only 60-75% accuracy on diverse traffic. Classifier-based routing uses a trained model like fine-tuned BERT or logistic regression over embeddings with 3-10ms latency and 88-94% accuracy. LLM-based routing uses a small model like GPT-4o-mini to evaluate complexity with 200-500ms latency but the highest accuracy for novel query types. Cascade routing sends everything to the cheapest model first and escalates to more expensive models only when the verifier rejects the output, which avoids a routing decision entirely but may waste the cheap model's compute on complex queries. Load balancing strategies, model count categories, and timing patterns are different taxonomies that do not describe the routing decision architecture for compound AI systems.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
4

What is the cascade pattern in compound AI systems?

  • The cascade routes every request to the cheapest model first and escalates to more expensive models only when the verifier rejects the output, avoiding an upfront routing decision entirely
  • The cascade pre-processes the query through increasingly sophisticated NLP stages before the LLM call, adding entity extraction, sentiment analysis, and intent classification as sequential steps
  • The cascade runs all model tiers simultaneously on every request and returns whichever response completes first, optimizing for latency rather than cost across the compound pipeline
  • The cascade stores model outputs in a hierarchical cache where tier 1 outputs are checked first, then tier 2, then tier 3, returning the cheapest cached response that matches the query

Why

The cascade pattern avoids the upfront cost and latency of a routing classifier by sending every request to the cheapest model first. If the verifier accepts the output, the request is served at minimum cost. If the verifier rejects it, the request escalates to the next tier. The worst case is a query that fails at all cheap tiers and reaches the frontier model, paying for the cheap attempts plus the expensive one. The best case, which covers 60-80% of traffic, is a query handled correctly by the cheapest model with no wasted compute on routing. The cascade works well when the verifier is reliable and cheap, because each tier requires a verification call. It works poorly when verification is expensive or when most queries need frontier capability, because the cheap-tier attempts waste tokens. Running all tiers simultaneously is a different pattern called speculative execution. NLP pre-processing stages are a pipeline pattern, not a cascade. Hierarchical caching is a caching strategy, not a cascade routing pattern.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
5

What is a system contract in compound AI design, and what are its three parts?

  • A system contract defines the input schema each component expects, the output schema it guarantees, and the SLA it commits to for latency, availability, and cost, enabling components to be swapped independently
  • A system contract defines the business agreement between the AI provider and the customer, specifying pricing tiers, usage limits, and support levels for each model in the compound pipeline
  • A system contract defines the security permissions for each component, specifying which API keys it can access, which network endpoints it can reach, and which data stores it can read or write
  • A system contract defines the training data requirements for each model in the pipeline, specifying the minimum dataset size, label quality, and evaluation metrics for fine-tuning each component that accounts for all operational constraints in multi-component AI architectures

Why

A system contract has three parts that define how a component interacts with the rest of the pipeline. The input schema specifies the exact structure the component expects, defined as a Pydantic model with typed fields and constraints. The output schema specifies the exact structure the component produces, enabling downstream components to consume the output without parsing ambiguity. The SLA specifies performance commitments: P95 latency, availability percentage, and cost per call. The contract boundary is what makes components swappable: the pipeline only sees input-to-output transformation, and which model or implementation produces it is an internal detail. This enables independent model upgrades where swapping GPT-4o for Claude Sonnet requires no changes to the classifier or verifier. Contracts are versioned with semantic versioning, where new fields are always optional with defaults to prevent cascade failures. System contracts are not business agreements, training requirements, or security permissions.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
Free account

Take the full module

These are the first few of 30 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack