AI Agents in Production — Advanced

Why

The success probability for a multi-step agent plan where each step succeeds independently with probability p is p raised to the N power. At 97% accuracy with 10 steps, this is 0.97 to the 10th power, which equals approximately 0.737 or about 74%. Compare this to 95% accuracy at 10 steps: 0.95 to the 10th equals 0.60, or 60%. The 2-percentage-point improvement in per-step accuracy raises the 10-step success rate from 60% to 74%, a 14-percentage-point improvement. This demonstrates that small improvements in per-step reliability have outsized impact on multi-step plans because the improvement compounds at each step. The 85% estimate incorrectly assumes non-uniform error distribution across steps. The 60% estimate uses the old 95% accuracy rate, not the improved 97%. The 90% estimate overstates the improvement. The practical implication is that investing in better tool descriptions, structured outputs, and observation validation to push per-step accuracy from 95% to 97% can nearly double the viable plan length from 5 steps to 10 steps while maintaining acceptable success rates.

Why

Each iteration of the agent loop sends the full conversation history to the model as input. Iteration 1 sends just the system prompt and query. Iteration 2 sends the system prompt, query, plus the first assistant-observation round. Iteration N sends all N-1 prior rounds. The total input across N iterations is N times S plus N times Q plus R times the sum from 1 to N-1, which equals R times N times (N-1) divided by 2. With the system prompt repeated N times and the query repeated N times, plus the quadratic history term, the formula becomes N times (S plus Q) plus R times N times (N+1) divided by 2. Plugging in S=2000, Q=100, R=400, N=10: 10 times 2100 plus 400 times 10 times 11 divided by 2 equals 21000 plus 22000 equals 43000. The linear estimate of 25000 would assume each iteration adds a fixed amount without the growing history. The system prompt is not re-tokenized; it is the growing conversation history that creates the quadratic term. Output lengths do not systematically grow per iteration. Tool result sizes are typically consistent. This quadratic growth is why prompt caching, history truncation, and cheaper models for routing steps are critical cost optimizations.

Why

Before MCP, each agent framework implemented its own tool interface: LangChain tools are Python classes with a _run method, OpenAI plugins use an OpenAPI spec, and custom agents use ad-hoc function registries. A tool built for one framework had to be rewritten for another. MCP inverts this by defining a standard JSON-RPC interface that any server can implement and any client can consume. The server is written once; clients are interchangeable. This is the same design principle as REST APIs: define the interface, not the implementation. MCP tools do not run inside the inference process; they run as separate server processes communicating over standard transports. Tool definitions are not compiled into the system prompt; they are discovered at runtime through the protocol's capability negotiation. MCP does not provide built-in auth or rate limiting; those are concerns of the transport and application layers. The practical benefit is tool ecosystem reuse: an MCP server for database access works with Claude Desktop, Cursor, a LangGraph agent, or any custom MCP client without any adapter code.

Why

For production agents handling 1000 concurrent conversations, PostgresSaver is the correct choice for three reasons. First, durability: MemorySaver stores checkpoints in a Python dictionary that is lost when the process restarts, meaning all active conversations lose their state on any restart or deployment. Second, multi-process access: SQLite uses file-level locking that does not support concurrent writes from multiple application processes, which is required when running multiple API server instances behind a load balancer. Postgres handles concurrent writes through its MVCC transaction system. Third, operational maturity: Postgres provides built-in replication, backups, and connection pooling. The 12ms write latency for PostgresSaver versus 0.02ms for MemorySaver is negligible when LLM calls in the agent loop take 500-2000ms per step; the checkpointer latency is less than 1% of total step time. The checkpoint backend absolutely affects normal operation because it determines whether conversations can be resumed after process restarts and whether multiple server instances can serve the same conversation. Co-locating the database with the agent service minimizes the network round-trip that dominates Postgres write latency.

Why

Injecting all accumulated long-term memories into the system prompt creates two problems that scale with memory count. First, cost: 500 facts at 20 tokens each adds 10,000 tokens to every prompt. At GPT-4o's $2.50 per million input tokens, this adds $0.025 per call. For an agent making 5 calls per task across 10,000 daily tasks, that is $1,250 per day just for memory tokens. Second, attention degradation: as the prompt grows longer, the model's effective attention on the user's actual query and task instructions decreases, leading to less relevant and less accurate responses. Retrieval solves both problems by embedding past interactions and searching for only the 5-10 most relevant memories at query time, keeping the injected context small and highly relevant. The tokenizer does not have a buffer limit on semantic units; it processes the full prompt. The Store API does not have a 500-entry capacity limit. While attention degradation is real, it manifests as reduced relevance and accuracy, not specifically as confabulation between unrelated facts. The hybrid approach is standard: inject the 5-10 most critical persistent facts directly and retrieve situationally relevant memories on demand.