Nybble™
← Back to modules

Self-Improving AI Systems — Advanced

Improvement cycle economics, DSPy search spaces, canary statistics, shadow evaluation design, rollback calibration, prompt version branching, meta-agent security, failure extraction costs, principle extraction, topology evolution economics, memory retrieval trade-offs, governance audit design.

Hard30 questions

Sample questions

1

A daily improvement cycle analyzes 500 traces, generates 3 candidate prompts, and evaluates each against a 50-example test set. At GPT-4o rates, what is the approximate cost per cycle?

  • About $50-100, because each candidate must be evaluated against the full 500-trace dataset to ensure statistical significance across the complete distribution of production queries
  • About $500-1000, because shadow evaluation runs both the current and candidate prompts on all production traffic during the cycle to collect paired quality comparisons
  • About $0.10-0.30, because the meta-agent reuses cached LLM responses from the production agent's earlier calls and only pays for the incremental tokens in the analysis prompt
  • About $3-8, computed as trace analysis ($0.20-0.50) plus 3 candidates at $0.30-0.50 generation each plus 3 evaluations at $0.50-1.00 each on the 50-example test set

Why

The improvement cycle has three cost components. Trace analysis reads 100 failure traces and uses an LLM to categorize them by root cause, costing approximately $0.20-0.50 in API calls. Candidate generation produces 3 prompt revisions, each requiring one LLM call to analyze the failure patterns and propose changes, costing approximately $0.30-0.50 per candidate. Evaluation runs each candidate against a 50-example test set, making 50 LLM calls per candidate at the production model's rate, costing approximately $0.50-1.00 per candidate. Total: $0.20-0.50 plus 3 times $0.30-0.50 plus 3 times $0.50-1.00 equals approximately $3-8 per cycle. The $50-100 estimate incorrectly assumes evaluation against the full 500-trace dataset rather than the held-out 50-example test set. The $0.10-0.30 estimate incorrectly claims cached responses are reused. The $500-1000 estimate incorrectly includes shadow evaluation, which is a separate testing strategy not part of every improvement cycle. For a system spending $500 per day on production LLM calls, $3-8 adds only 0.6-1.6% overhead.

Building effective agents (Anthropic)DSPy documentation
2

DSPy's optimizer searches the space of instruction text and few-shot demonstrations. If there are 5 candidate instructions and 10 candidate demonstrations with selection of 3 demonstrations per prompt, what is the search space size?

  • 15 combinations, computed as 5 instructions plus 10 demonstrations, because DSPy evaluates each parameter independently and selects the best of each category separately
  • 50 combinations, computed as 5 instructions times 10 demonstrations, because each instruction is paired with each individual demonstration for a complete pairwise evaluation
  • 30 combinations, computed as 5 instructions times the 6 distinct orderings of 3 demonstrations chosen from 3 fixed slots, because demonstration order affects model performance significantly, which is the standard recommendation for production self-improving systems
  • 600 combinations, computed as 5 instructions times C(10,3)=120 demonstration subsets, because the optimizer must consider every possible combination of instruction and demonstration selection

Why

The search space is the Cartesian product of the instruction candidates and the demonstration subset candidates. With 5 candidate instructions and C(10,3)=120 ways to choose 3 demonstrations from 10 candidates, the total space is 5 times 120 equals 600 combinations. In practice, DSPy does not exhaustively evaluate all 600; it uses optimization strategies like random search, Bayesian optimization, or evolutionary algorithms to efficiently sample the space. The 15 estimate incorrectly adds parameters that are multiplicative, not additive. The 50 estimate pairs each instruction with individual demonstrations rather than subsets of 3. The 30 estimate incorrectly limits the demonstration space. The combinatorial explosion is why DSPy's optimization is valuable: manually testing even a fraction of 600 combinations would take days, while the optimizer explores the space systematically in minutes at $2-10 in API calls. The optimizer often finds non-obvious combinations that outperform human-selected prompts because it explores phrasing and example combinations a human would not try.

DSPy documentationPrompt Engineering Guide
3

A canary deployment at 5% of 10,000 daily requests runs for 4 hours. How many canary requests are available for regression detection, and what regression size can be detected with 95% confidence?

  • About 2,000 canary requests, sufficient to detect a 1% regression with 95% confidence because 4 hours covers 80% of daily traffic and the 5% canary slice receives proportional volume
  • About 83 canary requests, sufficient to detect roughly a 10% regression with 95% confidence, because 4 hours is one-sixth of a day and 5% of that fraction yields a small but statistically useful sample
  • About 10 canary requests, insufficient for any statistical analysis because 4 hours at 5% traffic produces too few data points for meaningful regression detection at any confidence level
  • About 500 canary requests, sufficient to detect a 3% regression because the 5% canary slice is measured over the full 24-hour day and only the 4-hour measurement window limits statistical power and has been validated across diverse production self-improvement deployments

Why

With 10,000 daily requests uniformly distributed, 4 hours represents one-sixth of the day, yielding approximately 1,667 total requests during that window. At 5% canary traffic, roughly 83 requests hit the canary version. With 83 observations, the minimum detectable effect at 95% confidence and 80% power for a proportion test is approximately 10 percentage points. This means a canary at 5% traffic for 4 hours can detect a dramatic regression like accuracy dropping from 85% to 75% but cannot detect a subtle 2-3% regression. The 2,000 estimate overestimates by ignoring the time window. The 500 estimate confuses the canary slice timing. The 10 estimate underestimates the sample size. The practical implication is that canary windows should be long enough to accumulate sufficient data: 24 hours at 5% yields approximately 500 requests, sufficient to detect a 5% regression. For faster detection of smaller regressions, increase the canary percentage from 5% to 25%.

Building effective agents (Anthropic)DSPy documentation
4

Shadow evaluation runs both old and new prompts on every request, returning only the old prompt's response. What is the cost multiplier, and when is it preferred over canary deployment?

  • The cost multiplier is 2x because every request makes two LLM calls, and it is preferred for high-stakes applications where any user-facing quality regression is unacceptable
  • The cost multiplier is 1.5x because the shadow call uses a cheaper model than the production call, and it is preferred when the production model is too expensive to run twice at full price
  • The cost multiplier is 3x because shadow evaluation also runs a third LLM-as-judge call to compare the two responses, and it is preferred when automated quality scoring is more reliable than user feedback
  • The cost multiplier is 1x because shadow calls are free since they use cached responses from the production call's key-value attention states without running a separate inference

Why

Shadow evaluation makes two full LLM calls per request: one with the active prompt whose response is served to the user, and one with the candidate prompt whose response is logged but never shown. This doubles the LLM spend during the shadow period. The 2x cost is justified for high-stakes applications like medical, financial, or legal systems where exposing even 5% of users to a potentially worse prompt is unacceptable. Shadow evaluation introduces zero risk to users because they only see the active version's response. An LLM-as-judge can then score both responses on the same input, producing a win/loss/tie ratio after 200-500 paired comparisons. The 1.5x estimate incorrectly assumes a cheaper shadow model. The 1x estimate incorrectly claims cache reuse; the shadow call runs full inference with a different prompt. The 3x estimate incorrectly includes a mandatory judge call as part of the cost multiplier; judging is optional and typically runs on a sample. For consumer applications where a 5% quality dip on 5% of users is acceptable, canary is faster and cheaper at 1x cost.

Building effective agents (Anthropic)DSPy documentation
5

A rollback trigger monitors canary quality with a 30-minute window and a 75% accuracy threshold. The canary's true accuracy is 78% but measurement noise causes readings of 72-84%. What is the risk?

  • The noise is irrelevant because the 30-minute window provides enough data points to average out any measurement variance and produce a reliable accuracy estimate
  • The threshold should be raised to 80% to provide a safety margin above the true accuracy, ensuring that any reading below 80% triggers a rollback even if the regression is small, based on empirical data from systems running automated improvement loops, based on empirical data from automated improvement loops in production
  • The monitoring window is too long and should be shortened to 5 minutes to catch regressions faster, since waiting 30 minutes exposes users to a degraded experience unnecessarily
  • The 75% threshold is too close to the noise floor, causing false rollbacks when random variance dips the measured accuracy below 75% even though true accuracy exceeds it, while setting the threshold at 70% would reduce false triggers without missing genuine regressions

Why

With true accuracy at 78% and noise producing readings from 72% to 84%, the 75% threshold falls within the noise range. Random variance will occasionally produce a reading below 75% even though the prompt is performing acceptably, triggering a false rollback. The fix is to lower the threshold to a value below the noise floor: at 70%, the threshold is safely below the 72% lower bound of the noise range, so only genuine regressions (true accuracy dropping below 70%) trigger rollbacks. The noise is not irrelevant; 30 minutes of canary traffic at typical volumes yields a small sample where measurement variance is significant. Shortening the window to 5 minutes makes the problem worse because fewer data points increase variance. Raising the threshold to 80% would cause even more false rollbacks because the true accuracy of 78% regularly measures below 80% due to noise. The monitoring window should be long enough to accumulate sufficient data: too short means high variance and false triggers, too long means slow response to genuine regressions. The threshold must be set below the expected noise floor of acceptable performance.

Building effective agents (Anthropic)DSPy documentation
Free account

Take the full module

These are the first few of 30 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack