Nybble™
← Back to modules

Compound AI Systems — Advanced

Multi-tier routing economics, cascade vs classifier trade-offs, verifier ROI, semantic cache tuning, shadow testing statistics, contract versioning, AI gateway composition, agent registry discovery, fallback chain design, classifier training methodology, observability span correlation, cost optimization formulas.

Hard30 questions

Sample questions

1

A system handles 100K requests/day. Baseline frontier cost is $3,000/day. With 70% routed to small models ($77/day), 30% on frontier ($900/day), and classifier overhead ($5/day), what is the daily cost and percentage savings?

  • About $1,500/day with 50% savings, because the routing split averages the frontier and small model costs equally across the full volume of production traffic
  • About $500/day with 83% savings, because semantic caching handles an additional 50% of the small-model traffic at zero incremental LLM cost on top of the routing savings
  • About $982/day with 67% savings, computed as $77 small-model cost plus $900 frontier cost plus $5 classifier overhead, confirming the 40-70% savings range reported in production deployments
  • About $2,100/day with 30% savings, because the 30% of complex queries on frontier models dominate the cost and limit the achievable savings regardless of how much traffic is routed cheaply, based on empirical data from compound systems in production

Why

The calculation follows the routing cost formula directly. Small model daily cost for 70,000 requests at approximately $1/M input and $2/M output with 500 input and 300 output tokens per request equals roughly $77. Frontier model daily cost for 30,000 requests at $15/M input and $75/M output equals roughly $900. Classifier overhead for 100,000 requests at approximately 200 tokens each at $0.25/M equals roughly $5. Total daily cost is $77 plus $900 plus $5 equals $982. Savings from the $3,000 baseline are $2,018, or approximately 67%. This aligns with the 40-70% savings range reported in production deployments. The $1,500 estimate incorrectly averages costs. The $2,100 estimate overestimates frontier dominance. The $500 estimate incorrectly adds caching benefits not specified in the scenario. The classifier costs less than 1% of the total budget but controls where the other 99% goes. The break-even point for adding routing infrastructure is approximately $5,000 monthly inference spend.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
2

The cascade pattern sends every request to the cheapest model first and escalates on verifier rejection. Compared to upfront classifier routing, when does cascade perform better economically?

  • Cascade always performs better because it guarantees the minimum possible cost for every request by trying the cheapest option first, while classifier routing wastes the classifier's cost on every single request
  • Cascade performs better when the system has a single model provider because it eliminates the need for cross-provider routing logic that classifier-based systems require for fallback handling
  • Cascade performs better when over 60-70% of traffic can be handled by the cheapest model, because the wasted cost of cheap-model attempts on complex queries is offset by the savings from avoiding a classifier on all queries
  • Cascade performs better when query complexity is uniformly distributed because it processes an equal number of requests at each tier, while classifier routing works only when query complexity follows a skewed distribution, based on empirical data from compound systems in production

Why

The cascade pattern's economics depend on the fraction of traffic the cheapest model can handle. When 60-70% or more of requests succeed at tier 1, the cascade saves money because most requests pay only the tier 1 cost without any classifier overhead. The wasted cost on the 30-40% of requests that fail at tier 1 and must escalate is the tier 1 attempt plus the successful higher-tier attempt. With a classifier, every request pays the classifier cost plus the routed tier cost. The cascade wins when the classifier cost times total traffic exceeds the wasted tier 1 cost times the escalation fraction. If only 30% of traffic can be handled by the cheap model, the cascade wastes 70% of tier 1 attempts, making classifier routing cheaper because the classifier correctly avoids tier 1 for those requests. The cascade does not always perform better; it depends on the traffic distribution. It does not require uniform complexity distribution. Single versus multi-provider is a separate consideration. The cascade also requires a reliable and cheap verifier to detect when escalation is needed.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
3

A verifier's LLM-as-judge catches 15-25% of quality failures that schema-only verification misses. The judge costs approximately $0.0002 per call using Haiku. For a system processing 100K requests/day, what is the daily judge cost and is it justified?

  • About $20/day, which is not justified because the judge adds 200ms of latency to every request and the quality improvement is marginal compared to the user-perceived slowdown
  • About $2/day, which is always justified regardless of application type because the cost is negligible and the quality improvement provides a measurable lift in user satisfaction across all compound system deployments and has been validated across production compound AI deployments
  • About $20/day, which is justified because catching 15-25% of quality failures that schema checks miss prevents 15,000-25,000 bad responses per day from reaching users, and each bad response has a trust cost that far exceeds $0.0002
  • About $200/day, which is justified only for enterprise customers with SLAs requiring 99.9% response quality, but prohibitively expensive for consumer applications where some quality variance is acceptable

Why

The daily LLM-judge cost is 100,000 requests times $0.0002 per call equals $20 per day, or approximately $600 per month. Whether this is justified depends on the cost of a bad response reaching the user. If the system handles customer support where a wrong answer causes a support ticket costing $5-50 to resolve, catching 15,000-25,000 bad responses per day avoids $75,000-1,250,000 in downstream costs. Even for lower-stakes applications, the $20 daily cost is typically less than 1% of the total inference budget. The judge can be selectively applied: run it only on tier 1 and tier 2 outputs where errors are more likely, skipping frontier-tier outputs where the quality baseline is higher. This reduces the judge cost to $14-17 per day. The $200 estimate overestimates by 10x. The $2 estimate underestimates by 10x. The latency argument about 200ms is valid for the affected requests but does not make the quality improvement marginal. The judge adds latency but catches real errors.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
4

A semantic cache with threshold 0.90 returns wrong answers because 'What is Python?' matches 'What is a python snake?' Raising the threshold to 0.95 fixes this but reduces the hit rate from 45% to 30%. What is the economic trade-off?

  • The trade-off is purely about accuracy: 0.95 eliminates all false matches while 0.90 introduces errors, and accuracy should always take priority over cache hit rate regardless of the cost implications
  • At 0.95, the 15-percentage-point lower hit rate means 15,000 more LLM calls per 100K requests, costing approximately $150-750 more per day, but this is justified because each false cache hit at 0.90 returns a definitively wrong answer that damages user trust
  • Lowering the threshold to 0.85 would be optimal because it maximizes the hit rate at 55%, and the additional false matches can be caught by the verifier component before reaching the user
  • The trade-off is negligible because both thresholds produce approximately the same quality outcomes, and the difference in hit rate primarily affects latency rather than cost since cache lookups are free and has been validated across production compound AI deployments

Why

The economic trade-off involves comparing the cost of additional LLM calls against the cost of false cache hits. At 0.90 threshold with 45% hit rate on 100,000 daily requests, 45,000 requests are served from cache at zero LLM cost. At 0.95 with 30% hit rate, only 30,000 are cached, meaning 15,000 additional requests require fresh LLM calls. At $0.01-0.05 per LLM call depending on the model tier, the additional cost is $150-750 per day. However, each false cache hit at 0.90 returns a definitively wrong answer: the user asked about the programming language Python but received information about snakes. This is worse than a cache miss because it presents incorrect information with full confidence. The damage to user trust from wrong cached answers is worth far more than $150-750 per day. The accuracy-first argument is directionally correct but does not quantify the trade-off. The negligible-difference claim is wrong since the quality difference is significant. Lowering to 0.85 would increase false matches further. Starting at 0.95 and lowering only after measuring false positive rates on the specific distribution is the standard approach.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
5

Shadow testing runs both models on every request at 2x LLM cost. How many paired comparisons are needed to make a statistically reliable deployment decision?

  • 50-100 pairs, sufficient to detect a 15-20% quality difference with 80% power, which is the minimum for making any deployment decision about a model swap
  • 200-500 paired comparisons scored by an LLM-as-judge, producing a win/loss/tie ratio with sufficient statistical power to distinguish a genuine 3-5% quality improvement from random variation
  • 10,000+ pairs, because compound AI systems serve diverse traffic and the shadow test must cover every query category with statistical significance independently before a system-wide deployment
  • 1,000-5,000 pairs, required because LLM-as-judge scores have high variance and smaller samples produce confidence intervals too wide for practical decision-making about production model swaps

Why

Shadow evaluation produces paired outputs where both the current and candidate models respond to the same query. An LLM-as-judge scores each pair, producing a win/loss/tie outcome. After 200-500 paired comparisons, the win/loss/tie ratio has sufficient statistical power to distinguish a genuine 3-5% quality improvement from random variation. The signal becomes clear: if the candidate wins 55% of paired comparisons, ties 30%, and loses 15%, that is a strong positive signal across hundreds of pairs. Below 200 pairs, confidence intervals are too wide to make reliable decisions. Above 500, the additional statistical precision rarely changes the decision. At 2x LLM cost during the shadow period, 500 pairs at $0.01-0.05 per pair costs $10-50 in additional LLM spend: a negligible amount compared to the daily production budget. The 50-100 range can only detect very large differences. The 1,000-5,000 range overestimates the required sample size for paired comparisons where each pair is its own control. The 10,000+ range is unnecessary for system-level decisions, though per-category analysis within the 500 pairs can identify category-specific regressions.

The shift to Compound AI Systems (BAIR)Building effective agents (Anthropic)
Free account

Take the full module

These are the first few of 30 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack