Key takeaways

The through-line

State-of-the-art systems are not one model — they are routers, verifiers, caches, gateways and several models with different costs and strengths. Compound AI is the engineering discipline of composing them so the system is better, cheaper and more reliable than its best single component.

The ideas that carry

• One model is a single point of quality, cost and failure. Composition lets you spend capability only where the task needs it.
• Classifier → generator → verifier is the workhorse triad. Classify to route, generate to produce, verify to catch. The verifier is the piece most often skipped and most often the difference between demo and production.
• Components need contracts. Typed inputs and outputs at every boundary are what make a compound system debuggable instead of an emergent mess.
• Routing is where the money is. Send the easy majority to a small model and escalate the rest; most workloads are dominated by requests that never needed the frontier model.
• Cascades and fallbacks turn provider incidents into latency, not outages.
• A gateway centralises the cross-cutting concerns — auth, rate limits, retries, logging, budgets — that otherwise get reimplemented badly in every service.
• Semantic caching is a large, sharp lever. Big savings and a real risk of serving a near-miss answer; the similarity threshold is a product decision, not a tuning detail.
• MCP and A2A are the protocol stack — MCP for tools, A2A for agent-to-agent — and registries are what make capability discovery possible beyond hardcoded lists.
• Observability must span components. Per-call traces don't explain a system; you need the whole path, its cost and where quality was lost.

What you can now do

Decompose a task into routed components with real contracts. Cut cost substantially without losing quality. Shadow-test a model swap before it reaches users. Trace a request end-to-end across a compound system.

If one thing sticks

Ask "what is the cheapest component that can do this step correctly?" at every stage. Compound systems win on that question, not on having the best model.