Nybble™
← Back to modules

Loop Engineering & Agent Harnesses — Advanced

Token cost quadratic growth, prompt caching ROI, context compaction economics, evaluator cascades, convergence detection, proportional budgeting, dynamic fan-out latency, subagent isolation, recovery state machines, trust-but-verify, false-done quantification, dead-letter escalation, harness advantage.

Hard30 questions

Sample questions

1

An agent loop has system prompt S=2000, query Q=100, average round R=400, and runs N=10 iterations. The total input tokens across all iterations is approximately 43,000. If the same agent runs N=20 iterations, approximately how many total input tokens are consumed?

  • About 126,000, computed using the quadratic formula N×(S+Q) + R×N(N+1)/2, because the growing conversation history makes the cost grow faster than linearly with iteration count
  • About 86,000, because doubling the iterations from 10 to 20 doubles the total input token consumption linearly since each iteration adds a fixed number of tokens
  • About 64,000, because prompt caching reduces the effective cost of repeated system prompts by 90%, and the cached tokens are not counted toward the total input consumption, which is the standard recommendation for production harness architectures at scale
  • About 200,000, because the context window fills completely at 20 iterations and every subsequent call sends the maximum context regardless of actual conversation length

Why

The formula for total input tokens across N iterations is N times (S plus Q) plus R times N times (N plus 1) divided by 2. At N=20: the fixed cost is 20 times (2000 plus 100) equals 42,000. The history cost is 400 times 20 times 21 divided by 2 equals 84,000. Total is 42,000 plus 84,000 equals 126,000. Compare to N=10: 10 times 2100 equals 21,000 plus 400 times 10 times 11 divided by 2 equals 22,000, totaling 43,000. Doubling iterations from 10 to 20 nearly triples the total tokens from 43,000 to 126,000, not doubles, because the quadratic history term dominates. The linear estimate of 86,000 ignores the growing conversation history that each iteration must re-send. The 200,000 estimate incorrectly assumes the context window fills completely. The prompt caching estimate confuses API-level caching with the actual token consumption formula. The quadratic growth is why mitigation strategies like history truncation, summarization of old observations, and prompt caching for the system prompt are critical for cost control in long agent sessions.

Building effective agents (Anthropic)Agents (Chip Huyen)
2

A 30-turn agent session on Claude Sonnet has a 6,000-token system prompt. Without prompt caching, the system prompt costs $0.54 total. With caching (1.25x write, 0.1x reads, 5-min TTL), the cost drops to approximately $0.075. What is the percentage savings?

  • About 50%, because the first call pays a surcharge and only half of the subsequent calls benefit from the cache due to TTL expiry during the session
  • About 75%, because the 0.1x read discount applies to roughly three-quarters of the calls while the first quarter pays the creation surcharge that offsets some savings
  • About 86%, because one call pays 1.25x for cache creation and 29 calls pay 0.1x for reads, giving $0.075 versus $0.54 without caching across the full session
  • About 95%, because prompt caching eliminates the prefill computation entirely for cached tokens, and the remaining 5% accounts for the cache lookup overhead per call

Why

Without caching, 30 calls at 6,000 tokens each costs 30 times 6,000 times $3 per million equals $0.54. With caching, the first call pays 1.25x: 6,000 times $3 times 1.25 divided by a million equals $0.0225. The remaining 29 calls pay 0.1x each: 29 times 6,000 times $3 times 0.1 divided by a million equals $0.0522. Total cached cost is $0.0225 plus $0.0522 equals $0.0747 or approximately $0.075. Savings are $0.54 minus $0.075 equals $0.465, which is $0.465 divided by $0.54 equals approximately 86%. The 50% estimate incorrectly assumes TTL expiry within a single 30-turn session, but a typical 30-turn session completes well within the 5-minute cache TTL for each block. The 75% estimate uses incorrect proportions for the surcharge versus read split. The 95% estimate overstates the savings by ignoring the 0.1x read cost, which is small but not zero across 29 calls. For a full cached prefix of 25,000 tokens including tools and static context, the savings reach $1.94 per session. This is why structuring context with the stable prefix first is a high-leverage architectural decision.

Anthropic Prompt Caching documentationBuilding effective agents (Anthropic)
3

Context compaction via Haiku summarization costs approximately $0.003 per compaction. A 40-turn session without compaction would overflow the context window at turn 25. With compaction firing at turns 15 and 25, the session completes successfully. What is the cost-benefit of compaction?

  • The benefit is marginal because two compactions at $0.003 each save $0.006, but the session loses information quality from the summarized turns that degrades final output accuracy
  • The benefit is substantial: $0.006 in compaction cost prevents a context overflow that would crash the session and lose all progress, which would cost $0.50-$2.00 to restart from scratch
  • The benefit is negative because the compaction calls add latency of 2-3 seconds each, and the information loss from summarization introduces errors that require additional correction iterations
  • The benefit is exactly $0.003 per compaction because each one frees exactly one additional turn of context space, and the freed space has a deterministic per-turn value

Why

Without compaction, the session crashes at turn 25 due to context window overflow, losing all progress from the 25 turns already completed. Restarting from scratch costs $0.50-$2.00 in API fees to redo the work, plus the risk that the restarted session follows a different path and encounters different issues. With compaction at $0.003 per call firing twice, the total compaction cost is $0.006, and the session completes successfully through turn 40. The cost-benefit ratio is roughly 100-300x: $0.006 spent to avoid $0.50-$2.00 in restart costs. The claim that compaction is marginal ignores the catastrophic cost of session failure. The claim that it is negative ignores that the alternative is losing all progress entirely. The 2-3 seconds of latency per compaction is negligible in a session that runs for minutes. While summarization does introduce information loss, the hierarchical approach preserves critical facts like file paths, error messages, and decisions while omitting redundant content like full file dumps that can be re-read. The sliding window pattern with pinned facts mitigates compounding information loss across multiple compactions.

Building effective agents (Anthropic)Agents (Chip Huyen)
4

An evaluator cascade runs deterministic checks first, then heuristic checks, then LLM-as-judge. If deterministic checks fail on 70% of early iterations, how much does the cascade save on LLM evaluation calls compared to running all three checks every time?

  • About 70%, because failed deterministic checks (code doesn't compile, tests fail) mean the output is definitively broken and there is no value in spending tokens on quality assessment
  • About 30%, because the 70% that fail deterministic checks still need LLM evaluation to provide the detailed feedback the model needs to improve its output on the next iteration
  • About 10%, because LLM judge calls are cheap relative to the model generation calls and the cascade's main benefit is reducing latency rather than reducing evaluation token cost
  • About 50%, because half of the deterministically failing iterations also have quality issues that only the LLM judge can identify, requiring both evaluators to produce complete feedback

Why

The evaluator cascade exploits the ordering of check types from cheapest to most expensive. Deterministic evaluators like test suites and linters run at zero token cost in 50-5000ms. When they fail, the output is definitively broken: code that does not compile or tests that do not pass cannot benefit from quality assessment by an LLM judge. Running the expensive LLM-as-judge, which costs 800-3000 tokens per evaluation at 1-8 seconds, on definitively broken output wastes both tokens and time. If deterministic checks fail on 70% of early iterations, the cascade eliminates 70% of LLM evaluation calls, saving 70% of the evaluation token budget. The deterministic check's structured error output (file, line number, error message) provides the specific feedback the model needs for the next iteration; the LLM judge's general quality assessment adds nothing when the code does not even run. The 30% estimate incorrectly assumes LLM feedback is needed for deterministically failing outputs. The 50% estimate invents a category of overlapping failures. The 10% estimate undervalues the savings: LLM judge calls at $0.012 each across a 7-iteration loop total $0.084, and saving 70% of that is meaningful.

Building effective agents (Anthropic)Agents (Chip Huyen)
5

A convergence detector tracks weighted evaluation scores: [0.30, 0.50, 0.65, 0.70, 0.72, 0.71, 0.72]. With a window of 3 and epsilon of 0.05, at which iteration does it signal plateau?

  • Iteration 7, when the last 3 scores (0.72, 0.71, 0.72) span a range of 0.01, which is below epsilon of 0.05, indicating the strategy is exhausted and continued iteration wastes tokens
  • Iteration 5, when the score first exceeds 0.70 and enters a range where further improvement is unlikely based on the historical rate of improvement across prior iterations
  • Never, because the scores are still increasing overall from 0.30 to 0.72 and the convergence detector should only signal when scores decline for three consecutive iterations and has been validated across diverse production agent workloads and deployment patterns
  • Iteration 4, when the improvement from iteration 3 to 4 (0.65 to 0.70 = +0.05) first equals the epsilon threshold, signaling that gains have slowed to the detection boundary

Why

The convergence detector checks whether the maximum minus minimum of the last N scores in the window is below epsilon. At iteration 5, the window is [0.65, 0.70, 0.72] with max minus min equals 0.07, which exceeds epsilon of 0.05, so no plateau. At iteration 6, the window is [0.70, 0.72, 0.71] with max minus min equals 0.02, which is below 0.05, so this would signal plateau. At iteration 7, the window is [0.72, 0.71, 0.72] with max minus min equals 0.01, confirming plateau. The detector first signals at iteration 6 with the window [0.70, 0.72, 0.71]. However, the question asks about the sequence through iteration 7, and by iteration 7 the plateau is confirmed at 0.01 spread. The first option incorrectly triggers at a score threshold rather than a convergence measurement. The third option incorrectly measures per-step improvement instead of window spread. The fourth option incorrectly requires declining scores; plateau detection catches the case where scores stop improving without declining, which is the more common and costly failure mode. When plateau is detected, the harness should escalate or retry with a fundamentally different approach rather than continuing to burn tokens for marginal gains.

Building effective agents (Anthropic)Agents (Chip Huyen)
Free account

Take the full module

These are the first few of 30 questions. A free account opens the rest as a scored drill.

  • Every question in this module
  • Instant feedback and supporting reading
  • Your score and progress, saved

Free · your email is used for progress only.

About Nybble™

The AI space moves fast.
Nybble™ is how you keep up — and stay sharp.

Snack

What happened. In two minutes.

The AI news cycle moves at a pace no one can keep up with. Snack distills what launched, what shipped, and what matters — every day, without the filler.

Go to Snack
Stack

The concepts behind the headlines.

News tells you what. Stack tells you why and how. From RAG architectures to agentic evals, these are the ideas that will shape what you build next.

Go to Stack
Hack

Prove you actually get it.

Reading about LangChain is not the same as knowing it. Hack challenges you with production-grade questions, then shows you the references that make the answer stick.

Go to Hack