The dual-loop architecture

The production agent processes a user request in 500ms-3s: receive query, retrieve context, call the LLM, parse the response, return the answer. This is the task loop — it runs on every request, optimized for latency, scoped to a single user interaction. If the agent fails on this request, the damage is one bad response.

The improvement loop operates on a different timescale entirely. It collects production logs over hours or days, extracts recurring failure patterns, generates hypotheses about what changes would fix them, tests those hypotheses against held-out data, and stages improvements for review. A single execution of the improvement loop might take 10-30 minutes and analyze 10,000 logged interactions. If the improvement loop produces a bad change and it passes review, the damage propagates to every subsequent request.

This asymmetry — fast but bounded damage versus slow but systemic damage — is why the two loops must be architecturally separated.

The task loop in detail

The task loop is what users interact with. Its structure is familiar: receive input, process through a pipeline of model calls and tool invocations, return output. The addition for self-improving systems is instrumentation — the task loop must emit structured logs that the improvement loop can consume.

import time
import uuid
import json
from dataclasses import dataclass, asdict
from openai import OpenAI

@dataclass
class ExecutionTrace:
    trace_id: str
    timestamp: float
    user_input: str
    system_prompt_version: str
    model: str
    raw_output: str
    parsed_output: dict
    latency_ms: float
    input_tokens: int
    output_tokens: int
    tool_calls: list[dict]
    quality_signals: dict

class TaskLoop:
    def __init__(self, prompt_registry, log_store):
        self.client = OpenAI()
        self.prompt_registry = prompt_registry
        self.log_store = log_store

    def execute(self, user_input: str, context: dict = None) -> dict:
        trace_id = str(uuid.uuid4())
        start = time.time()

        active_prompt = self.prompt_registry.get_active()

        messages = [
            {"role": "system", "content": active_prompt["content"]},
            {"role": "user", "content": user_input},
        ]

        response = self.client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            temperature=0,
        )

        latency_ms = (time.time() - start) * 1000
        result = response.choices[0].message.content
        usage = response.usage

        trace = ExecutionTrace(
            trace_id=trace_id,
            timestamp=time.time(),
            user_input=user_input,
            system_prompt_version=active_prompt["version"],
            model="gpt-4o",
            raw_output=result,
            parsed_output={"response": result},
            latency_ms=latency_ms,
            input_tokens=usage.prompt_tokens,
            output_tokens=usage.completion_tokens,
            tool_calls=[],
            quality_signals={},
        )
        self.log_store.write(asdict(trace))

        return {"trace_id": trace_id, "response": result}

The ExecutionTrace captures everything the improvement loop needs: what prompt version was active, what the model produced, how long it took, how many tokens it consumed, and — critically — a quality_signals field for downstream quality labels. Quality signals might come from explicit user feedback (thumbs up/down), implicit signals (did the user retry the query? did they edit the response?), or automated evaluators (an LLM-as-judge scoring the output).

The improvement loop in detail

The improvement loop runs on a schedule — daily, weekly, or triggered by quality metric degradation. It operates in five stages:

Stage 1: Log collection. Pull all execution traces since the last improvement cycle. Filter to traces with quality signals (not all traces will have feedback). A system processing 10,000 requests/day with a 5% feedback rate yields 500 labeled traces per day.

Stage 2: Pattern extraction. Cluster failures by root cause. The meta-agent reads a sample of low-quality traces and categorizes them: "fails on multi-part questions" (23% of failures), "hallucinates when context is insufficient" (18%), "wrong output format on edge cases" (12%). This step costs ~$0.20-0.50 in API calls for 100 failure traces.

Stage 3: Hypothesis generation. For each failure pattern, the meta-agent generates candidate fixes. For "fails on multi-part questions," it might propose: "Add instruction: 'For questions with multiple parts, address each part separately with a labeled section.'" For "hallucinates when context is insufficient," it might propose: "Add instruction: 'If the provided context does not contain enough information to answer, say so explicitly rather than inferring.'" Each hypothesis is a concrete prompt diff.

Stage 4: Testing on held-out data. Each candidate is evaluated against a held-out test set that was not used during hypothesis generation. The test set must include both examples of the failure pattern (to verify the fix works) and examples of currently-passing cases (to detect regressions). A candidate that fixes 80% of the targeted failures but regresses 5% of passing cases may not be worth deploying.

class ImprovementLoop:
    def __init__(self, meta_agent, prompt_registry, test_set: list[dict]):
        self.meta_agent = meta_agent
        self.prompt_registry = prompt_registry
        self.test_set = test_set

    def run_cycle(self) -> dict:
        current = self.prompt_registry.get_active()
        current_score = self.meta_agent.evaluate_candidate(current["content"])

        failures = self.meta_agent.analyze_failures(window_hours=24)
        if not failures:
            return {"action": "no_change", "reason": "no failure patterns detected"}

        candidates = []
        for _ in range(3):
            candidate = self.meta_agent.generate_candidate(failures)
            score = self.meta_agent.evaluate_candidate(candidate["improved_prompt"])
            candidates.append({**candidate, "score": score})

        best = max(candidates, key=lambda c: c["score"]["accuracy"])
        improvement = best["score"]["accuracy"] - current_score["accuracy"]

        if improvement < 0.02:
            return {
                "action": "no_change",
                "reason": f"best candidate only +{improvement:.1%} over current",
                "current_score": current_score,
                "best_candidate_score": best["score"],
            }

        return {
            "action": "propose",
            "candidate": best,
            "current_score": current_score,
            "improvement": improvement,
            "review_required": True,
        }

Stage 5: Staged deployment. If the best candidate passes the improvement threshold (typically 2-5 percentage points above the current version), it is staged for human review. The output is a diff showing what changed, the test results, and the failure patterns it addresses. A human approves or rejects the change. Approved changes enter canary deployment — 5% of traffic initially, scaling to 100% if production metrics hold.

The separation principle

The meta-agent that proposes improvements must be architecturally separate from the production agent. It runs in a different process, uses a different set of credentials, and has different access patterns:

Production agent: read access to user data (scoped per request), write access to response endpoints, no access to its own configuration.
Meta-agent: read access to production logs (aggregated, potentially anonymized), write access to the staging prompt registry, no access to user data, no access to production endpoints.

This separation is not just a security measure — it prevents feedback loops. If the production agent could modify its own prompt mid-request based on the current user's feedback, a single adversarial user could manipulate the system prompt for all subsequent users. The improvement loop's batched, offline, human-reviewed process is the firewall.

The review gate

Every proposed change surfaces as a structured diff for human review:

def format_review(proposal: dict) -> str:
    lines = [
        f"Improvement Proposal — {proposal['candidate']['created_at']}",
        f"Current accuracy: {proposal['current_score']['accuracy']:.1%}",
        f"Proposed accuracy: {proposal['candidate']['score']['accuracy']:.1%}",
        f"Improvement: +{proposal['improvement']:.1%}",
        "",
        "Changes made:",
        proposal["candidate"]["changes_made"],
        "",
        "Rationale:",
        proposal["candidate"]["rationale"],
        "",
        "Failure patterns addressed:",
    ]
    for pattern in proposal.get("failure_patterns", []):
        lines.append(f"  - {pattern['pattern']} ({pattern['frequency']} occurrences)")
    return "\n".join(lines)

The review gate is the control surface. In early deployments, every change goes through manual review. As confidence in the improvement loop grows, the threshold can shift: changes below 5% improvement require review; changes above 5% with no regressions can auto-deploy to canary. Fully automated deployment — no human in the loop — is the end state for mature systems, but most production deployments in 2026 keep the review gate active.

The cost structure of the dual-loop architecture: the task loop costs scale linearly with traffic (API calls per request). The improvement loop costs are fixed per cycle regardless of traffic — a daily cycle analyzing 500 traces and testing 3 candidates costs $3-8 in API calls. For a system spending $500/day on production LLM calls, the improvement loop adds 0.6-1.6% overhead.

← Previous