Nybble™
AI Agents in Production

Lesson 1 of 16

The agent loop

A chain executes a fixed sequence of steps: retrieve documents, stuff them into a prompt, call the LLM, parse the output. The execution path is determined at write time by the developer. An agent executes a variable sequence of steps: the LLM receives an observation, decides what action to take, executes it, observes the result, and decides again. The execution path is determined at runtime by the model. This distinction — who decides what happens next — is the defining boundary between chains and agents.

The core loop is Observe-Think-Act. The agent receives an observation (user query, tool output, error message), generates a thought (reasoning about what to do), selects an action (call a tool, return an answer), and the cycle repeats. Each iteration consumes the full conversation history as input tokens. A 5-iteration agent loop with a 4K-token system prompt and 500-token tool outputs processes roughly 5 * (4000 + 5 * 500) = 32,500 input tokens total across all calls. At GPT-4o's pricing of $2.50/M input tokens, that single query costs ~$0.08. A 15-iteration loop on a hard problem with long tool outputs can exceed $0.50 per query.

The ReAct pattern

ReAct (Yao et al. 2023) formalizes the Observe-Think-Act loop by interleaving reasoning traces with action execution. The model generates a Thought (free-text reasoning), then an Action (structured tool call), receives an Observation (tool output), and repeats until it produces a final answer. The key insight from the paper: models that reason before acting outperform models that only act (chain-of-action) and models that only reason (chain-of-thought without grounding). On HotpotQA, ReAct with GPT-3 achieved 34.3% exact match versus 29.4% for act-only and 28.7% for CoT-only.

The pattern constrains the model's output at each step to one of two formats: a thought-action pair (continue the loop) or a final answer (exit the loop). The parsing is mechanical — the framework looks for Action: and Action Input: markers to extract the tool call, or Final Answer: to terminate.

A minimal ReAct agent from scratch

This implementation uses no frameworks. The model is instructed to output structured markers, and Python string parsing extracts actions and routes them to tool functions.

import openai
import json
import re

client = openai.OpenAI()

TOOLS = {}

def tool(func):
    TOOLS[func.__name__] = func
    return func

@tool
def search(query: str) -> str:
    """Search Wikipedia for a topic. Returns a summary."""
    # Stub — in production, call the Wikipedia API
    data = {
        "Python": "Python is a programming language created by Guido van Rossum, released in 1991.",
        "Guido van Rossum": "Guido van Rossum is a Dutch programmer, creator of Python. Born January 31, 1956.",
    }
    for key, val in data.items():
        if key.lower() in query.lower():
            return val
    return f"No results for: {query}"

@tool
def calculate(expression: str) -> str:
    """Evaluate a mathematical expression."""
    try:
        result = eval(expression, {"__builtins__": {}})
        return str(result)
    except Exception as e:
        return f"Error: {e}"

SYSTEM_PROMPT = """You are a research agent. You solve tasks by interleaving Thought, Action, and Observation steps.

Available tools:
- search(query: str) — search for factual information
- calculate(expression: str) — evaluate a math expression

On each step, output EXACTLY one of:
1. A Thought/Action pair:
   Thought: <your reasoning>
   Action: <tool_name>
   Action Input: <argument>

2. A final answer:
   Thought: <your reasoning>
   Final Answer: <your answer>

Always start with a Thought. Never call two tools in one step."""


def parse_action(text):
    action_match = re.search(r"Action:\s*(\w+)", text)
    input_match = re.search(r"Action Input:\s*(.+?)(?:\n|$)", text, re.DOTALL)
    if action_match and input_match:
        return action_match.group(1), input_match.group(1).strip()
    return None, None

def parse_final_answer(text):
    match = re.search(r"Final Answer:\s*(.+?)$", text, re.DOTALL)
    return match.group(1).strip() if match else None


def run_agent(question, max_iterations=10):
    messages = [
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": question},
    ]

    for i in range(max_iterations):
        response = client.chat.completions.create(
            model="gpt-4o",
            messages=messages,
            temperature=0,
            max_tokens=512,
        )
        assistant_text = response.choices[0].message.content
        messages.append({"role": "assistant", "content": assistant_text})

        print(f"\n--- Iteration {i + 1} ---")
        print(assistant_text)

        final = parse_final_answer(assistant_text)
        if final:
            return final

        tool_name, tool_input = parse_action(assistant_text)
        if tool_name and tool_name in TOOLS:
            observation = TOOLS[tool_name](tool_input)
        elif tool_name:
            observation = f"Error: unknown tool '{tool_name}'"
        else:
            observation = "Error: could not parse action. Use the exact format."

        messages.append({"role": "user", "content": f"Observation: {observation}"})
        print(f"Observation: {observation}")

    return "Agent exceeded maximum iterations."


answer = run_agent("What year was the creator of Python born?")
print(f"\nFinal: {answer}")

This agent will typically take 2-3 iterations: search for Python, extract "Guido van Rossum," search for Guido, extract the birth year. Each iteration sends the growing conversation history to the model. By iteration 3, the input includes the system prompt, the original question, two assistant responses, and two observations — roughly 800 tokens. The total cost for this query at GPT-4o rates: ~$0.005.

Control flow: model-decided versus code-decided

The agent loop above gives the model full control over the execution path. The model chooses which tool to call, what arguments to pass, and when to stop. This is the maximum-autonomy configuration, and it is the most fragile.

In production, you almost always constrain the loop:

  • Maximum iterations — The max_iterations=10 parameter is a hard ceiling. Without it, a confused model can loop indefinitely, burning tokens. A 50-iteration loop on Claude 3.5 Sonnet at $3/M input tokens with 8K context can cost $1.20 on a single query.
  • Tool restrictions per step — Some tools should only be callable after others. A confirm_purchase tool should only be available after get_price has been called. This is code-decided control flow layered on top of model-decided tool selection.
  • Forced tool calls — The first step of a customer-support agent should always call lookup_customer, regardless of what the model thinks. OpenAI's tool_choice parameter and Anthropic's tool_choice: {"type": "tool", "name": "..."} force the model to call a specific tool.
  • Early termination — If the model's action would violate a business rule (deleting a production database, sending an email to the wrong domain), the code intercepts and terminates the loop before execution.

The practical spectrum runs from fully code-controlled (a chain — every step predetermined) to fully model-controlled (an autonomous agent — every step chosen by the LLM). Most production systems sit in the middle: the model decides within guardrails.

The autonomy-reliability tradeoff

Every iteration of the agent loop is a point where the model can make the wrong decision. If the model has a 95% chance of choosing the correct tool at each step, a 5-step plan succeeds with probability 0.95^5 = 0.77. A 10-step plan: 0.95^10 = 0.60. A 20-step plan: 0.95^20 = 0.36. Compound error rates are the fundamental reliability constraint on agents.

Strategies to improve per-step accuracy:

  • Better tool descriptions — The most impactful lever. Ambiguous descriptions like "search for stuff" cause wrong tool calls. Precise descriptions like "Search the product catalog by SKU or product name. Returns: product_id, name, price, stock_count" let the model match queries to tools reliably. Anthropic's documentation reports that improving tool descriptions from vague to precise can increase correct tool selection from ~80% to ~97% on internal benchmarks.
  • Fewer tools — An agent with 3 tools makes fewer selection errors than one with 30. If you have 30 tools, consider routing: a first-pass classifier selects a category, then a second agent operates within that category's subset of tools.
  • Structured outputs — Forcing the model to produce JSON conforming to a schema (OpenAI's response_format, Anthropic's tool use) eliminates parsing errors. The model cannot produce malformed tool calls if the output is schema-validated.
  • Observation validation — Check tool outputs before feeding them back. If a search returns empty results, inject a more helpful message than an empty string: "No results found. Try rephrasing or using a different tool."

Token economics of agent loops

Each iteration of the agent loop sends the full conversation history to the model. The cost is not linear — it is quadratic in the number of iterations because each call includes all prior iterations as input.

For an agent loop with system prompt S tokens, user query Q tokens, and average assistant+observation length R tokens per iteration, the total input tokens across N iterations is:

total_input = N * S + N * Q + R * N * (N + 1) / 2

The N * (N + 1) / 2 term comes from the growing history: iteration 1 has 0 prior rounds, iteration 2 has 1, iteration N has N-1. With S = 2000, Q = 100, R = 400, N = 10:

total_input = 10 * 2000 + 10 * 100 + 400 * 10 * 11 / 2 = 20000 + 1000 + 22000 = 43000 tokens

At GPT-4o pricing ($2.50/M input, $10/M output with ~200 output tokens per iteration):

cost = 43000 * 2.50/1e6 + 10 * 200 * 10/1e6 = $0.108 + $0.020 = $0.128

The same query on Claude 3.5 Sonnet ($3/M input, $15/M output): $0.159. On GPT-4 Turbo ($10/M input, $30/M output): $0.49. These costs are per query — a customer-support agent handling 10,000 queries/day at 5 iterations each would cost $640-$2,450/day in API fees on GPT-4o alone.

Mitigation strategies: truncate or summarize old observations before re-sending them, use cheaper models for simple routing steps (GPT-4o-mini at $0.15/M input is 17x cheaper than GPT-4o), or cache common tool-call patterns to skip the LLM entirely for known queries.