The Inner Agentic Loop

The Inner Agentic Loop

The inner loop is the cycle where the model receives context, produces a response, and optionally calls tools — repeated until a stop condition is met. The harness controls the loop; the model controls what happens within each iteration. This separation is critical: the harness owns the "when" and "whether"; the model owns the "what."

Immutable parameters vs. mutable state

The inner loop operates on two data structures that must remain architecturally distinct. Immutable parameters are set once before the loop starts and never change during execution:

• System prompt: the model's instructions, persona, and formatting rules. Typically 2,000-6,000 tokens. Cached aggressively.
• Tool definitions: JSON schemas describing available tools — name, description, input schema. Each tool definition costs 100-300 tokens. A set of 20 tools adds 4,000-6,000 tokens to every call.
• Model identifier: the specific model version (e.g., claude-sonnet-4-20250514, gpt-4o-2024-08-06). Pinning to a dated version prevents behavior regressions when providers update models.
• Temperature and sampling parameters: 0.0 for deterministic agent behavior in production. Non-zero only for creative generation tasks.
• Maximum output tokens per turn: 16,384 for Claude Sonnet, 4,096 for smaller tasks. This bounds the worst-case cost per iteration.
• Maximum total turns: a hard safety limit. Production systems typically set 50-100 turns. Without this, a confused model can loop indefinitely, burning budget.

Mutable state evolves with every iteration:

• Conversation history: growing list of messages (user, assistant, tool results). This is the primary memory mechanism.
• Tool execution results: appended after each tool call as user-role messages containing tool_result blocks.
• Pending task list: shrinks as subtasks complete (or grows if the model decomposes a task).
• Token counters: cumulative input tokens, output tokens, cache read tokens, cache creation tokens. These drive budget enforcement.
• Error counters: consecutive failures per category (tool errors, rate limits, parse failures). These trigger recovery policies at configured thresholds.
• Abort flag: set by external signals — timeout, user cancel, budget exhaustion. Checked at the top of every iteration.

Immutability of parameters is enforced in code, not by convention. In Python, use dataclass(frozen=True) or typing.NamedTuple. A bug that accidentally mutates the system prompt mid-loop — say, appending a tool result to it instead of to messages — is catastrophic. The model's behavior shifts unpredictably, and the bug is invisible in the output because the system prompt isn't included in response logs.

The iteration cycle in detail

Each pass through the inner loop executes these steps in strict sequence:

• Assemble context: concatenate system prompt + conversation history + pending tool results. Apply token budget constraints — if the assembled context exceeds budget, trigger compaction before the call. Token counting happens here using the model's tokenizer (or an approximation like tiktoken's cl100k_base for Claude-family models, which is accurate to within 2-3%).
• Normalize messages: ensure all messages conform to the provider's expected format. Claude expects alternating user/assistant turns; consecutive user messages must be merged. Tool results are user-role messages with tool_result content blocks, keyed by tool_use_id. OpenAI expects tool results as dedicated tool role messages. Normalization is a pure function: normalize(messages, provider) → messages. It handles provider-switching transparently.
• Call model: send the assembled messages to the model API. Set stream=True to enable early abort and incremental tool dispatch. Include cache breakpoints on the system prompt for providers that support it.
• Stream response: accumulate the response token-by-token. Parse tool-use blocks incrementally. Emit structured events: text_delta, tool_use_start, tool_input_delta, tool_use_complete, message_complete. Monitor for stop_reason in the final event.
• Execute tools: dispatch each tool call to the appropriate handler function. Enforce per-tool timeouts — file operations at 5 seconds, shell commands at 30 seconds, network requests at 15 seconds, code execution at 60 seconds. Capture the full result: stdout, stderr, return value, execution duration, exit code. Wrap execution in a try/except that converts all exceptions to structured error messages (never let a tool exception crash the loop).
• Check continue conditions: a priority-ordered list of exit checks:

- External abort signal fired → exit immediately

- Turn limit reached → exit with max_turns_exceeded

- Token budget exhausted → exit with budget_exceeded

- Model returned stop_reason: end_turn with no tool calls → exit with model_complete

- All pending tasks marked done → exit with tasks_complete

- None of the above → continue

• Loop or exit: if continuing, append the response and tool results to mutable state and restart from "assemble context." If exiting, return the final state to the outer harness.

import anthropic
from typing import Generator
import time

def inner_loop(
    client: anthropic.Anthropic,
    params: HarnessParams,
    state: HarnessState,
) -> Generator[dict, None, None]:
    
    while state.turn_count < params.max_turns:
        if state.abort_requested:
            yield {"type": "abort", "reason": "external_signal"}
            break
        
        token_count = count_message_tokens(state.messages)
        if token_count > params.token_budget:
            state.messages = compact_history(client, state.messages, params.token_budget * 0.7)
        
        messages = normalize_messages(state.messages, provider="anthropic")
        
        start_time = time.monotonic()
        response = client.messages.create(
            model=params.model,
            max_tokens=params.max_output_tokens,
            system=[
                {"type": "text", "text": params.system_prompt, "cache_control": {"type": "ephemeral"}},
            ],
            tools=params.tools,
            messages=messages,
        )
        latency_ms = (time.monotonic() - start_time) * 1000
        
        state.total_input_tokens += response.usage.input_tokens
        state.total_output_tokens += response.usage.output_tokens
        state.cache_read_tokens += getattr(response.usage, "cache_read_input_tokens", 0)
        state.turn_count += 1
        
        yield {
            "type": "model_response",
            "turn": state.turn_count,
            "input_tokens": response.usage.input_tokens,
            "output_tokens": response.usage.output_tokens,
            "latency_ms": latency_ms,
            "stop_reason": response.stop_reason,
        }
        
        assistant_message = {"role": "assistant", "content": response.content}
        state.messages.append(assistant_message)
        
        tool_calls = [b for b in response.content if b.type == "tool_use"]
        
        if not tool_calls:
            break
        
        tool_results = []
        for call in tool_calls:
            result = execute_tool_safe(call.name, call.input, timeout=30)
            tool_results.append({
                "type": "tool_result",
                "tool_use_id": call.id,
                "content": result.output if result.success else f"Error: {result.error}",
                "is_error": not result.success,
            })
            yield {"type": "tool_executed", "name": call.name, "success": result.success, "duration_ms": result.duration_ms}
        
        state.messages.append({"role": "user", "content": tool_results})
        
        if response.stop_reason == "end_turn":
            break

Recovery paths as explicit state machine branches

Recovery is not exception handling. Exception handling is reactive and unstructured — a try/except catches whatever happens and does its best. Recovery in a harness is proactive and deterministic: every known failure mode has a pre-defined response, and the failure mode is detected by inspecting structured return values, not by catching exceptions.

• Output token limit hit (stop_reason == "max_tokens"): the model's response was truncated at max_output_tokens. The harness detects this from the stop reason, triggers context compaction (summarize the oldest 60% of history to reclaim token budget), then appends a continuation message ("Your previous response was truncated. Continue from where you left off.") and retries. This is the only recovery path that retroactively modifies history. Cost: one compaction call (~$0.003 with Haiku) plus one retry at full price. Frequency: 5-10% of turns in long coding sessions with verbose file outputs.
• Tool execution failure: a tool returned an error — file not found, permission denied, timeout, malformed output. The harness appends the error as a tool result with is_error: true and lets the model decide how to proceed. After 3 consecutive failures on the same tool, the harness injects a system-level message: "Tool {name} has failed 3 consecutive times. Proceed without it or use an alternative approach." The model adapts — this is a prompt engineering technique, not a code branch. After a successful tool call, the failure counter resets.
• Rate limit (HTTP 429): exponential backoff starting at 1 second, doubling up to 64 seconds. The Retry-After header, when present, overrides the calculated delay. After 5 retries without success (total wait: ~2 minutes), switch to a fallback provider if configured (e.g., Claude → GPT-4o, or Sonnet → Haiku with a degraded-mode flag). Log the total wait time — at $200/hour engineering cost, 2 minutes of idle waiting costs $6.67 in developer time.
• Model error (HTTP 500, malformed JSON, empty response): retry once immediately — transient errors on model APIs occur at roughly 0.1-0.5% of calls depending on load. On second consecutive failure, log the full request payload (with message content hashed for privacy) and the raw response, then abort the current task with a diagnostic payload the outer harness can inspect.
• Context overflow: the assembled message list exceeds the model's maximum context window (200K for Claude Sonnet, 128K for GPT-4o). This should never happen if the context builder enforces budgets correctly on every iteration. If it does, it indicates a bug in token counting — log an alert, trigger emergency compaction (aggressive summarization to 50% of budget), and continue. The alert should page the on-call engineer because the token counter is unreliable.

@dataclass
class RecoveryPolicy:
    consecutive_tool_errors: dict[str, int] = field(default_factory=dict)
    rate_limit_retries: int = 0
    max_tool_retries: int = 3
    max_rate_limit_retries: int = 5
    
    def handle_tool_error(self, tool_name: str, error: str) -> str:
        self.consecutive_tool_errors[tool_name] = (
            self.consecutive_tool_errors.get(tool_name, 0) + 1
        )
        count = self.consecutive_tool_errors[tool_name]
        
        if count >= self.max_tool_retries:
            return "disable"
        return "retry_with_error"
    
    def handle_rate_limit(self, retry_after: float | None) -> float:
        self.rate_limit_retries += 1
        if self.rate_limit_retries > self.max_rate_limit_retries:
            return -1.0  # signal: switch provider
        if retry_after:
            return retry_after
        return min(2 ** self.rate_limit_retries, 64)
    
    def handle_max_tokens(self) -> str:
        return "compact_and_retry"
    
    def handle_model_error(self, attempt: int) -> str:
        if attempt == 1:
            return "retry_immediate"
        return "abort_with_diagnostic"
    
    def reset_tool_errors(self, tool_name: str):
        self.consecutive_tool_errors.pop(tool_name, None)

Thinking blocks and extended reasoning

When using Claude with extended thinking enabled (thinking: {"type": "enabled", "budget_tokens": 10000}), the response includes thinking blocks before content blocks. These contain the model's chain-of-thought reasoning. The inner loop handles them as follows:

• Thinking blocks are included in the assistant message appended to conversation history. They must be preserved for cache coherence — if you strip them, the next cache-read will miss because the prefix has changed.
• Thinking tokens count toward output token costs but are billed at a reduced rate ($3/M for Sonnet thinking tokens vs. $15/M for standard output tokens as of 2025).
• A typical thinking-enabled response for a complex coding task: 3,000-10,000 thinking tokens + 500-3,000 content/tool-use tokens. Total output cost per turn: 8000 * $3/1M + 2000 * $15/1M = $0.024 + $0.030 = $0.054.
• Extended thinking cannot be combined with temperature > 0 or with forced tool use (tool_choice: {"type": "tool", "name": "..."}) on Claude. The harness must validate parameter compatibility before entering the loop and fail fast with a clear error if incompatible settings are detected.

The inner loop is mechanical by design. It should be boring code — no clever abstractions, no inheritance hierarchies, no framework magic. The complexity lives in the recovery policies and the context assembly. The loop structure itself is a while True with a match on the current state.