Autonomous & Headless Agents — Advanced

Why

The probability of a given agent run being interrupted by a deployment is approximately agent_runtime divided by deploy_interval. With a 10-minute runtime and hourly (60-minute) deployments, this is 10 divided by 60 equals approximately 0.167 or 17%. This calculation assumes the agent run starts at a uniformly random time within the deployment interval. The 0.5% estimate drastically underestimates the overlap probability. The 50% estimate would require either a 30-minute runtime or 20-minute deployment intervals. The 5% estimate incorrectly assumes only one pod is affected per deployment, but Kubernetes rolling deployments cycle through all pods. With a 30-second grace period, an agent mid-execution receives SIGTERM and has 30 seconds to complete. If the in-flight LLM call takes 45 seconds, the pod is killed by SIGKILL when the grace period expires, losing all in-flight state. In a cluster with 50 agent workers and daily deployments, this hits 2-5 agents per deploy. Without durability via event sourcing or checkpointing, each interrupted run loses all completed work.

Why

The critical problem with replaying from scratch is that some tool calls are not safely repeatable. A GitHub comment posted twice creates duplicate comments visible to all issue participants. A deployment triggered twice causes a race condition between two rollouts. A database write applied twice may violate unique constraints or corrupt aggregate counters. A Slack message sent twice confuses the recipient. Journal-based replay solves this by returning the recorded result for each completed step without re-executing it, so no side effects are duplicated. The non-determinism argument about LLM outputs is valid but secondary; the primary concern is duplicated side effects, not divergent reasoning paths. API format changes between execution and replay are possible but rare within the typical crash-to-recovery window of minutes. Context window overflow from replay metadata is not a real concern because journal replay does not add metadata to the conversation; it transparently returns the recorded result. The journal approach is deterministic and cheap: replaying 50 steps takes 10-50ms at zero API cost.

Why

Temporal Workflows must be deterministic: given the same input and event history, they must produce the same sequence of commands. This is because replay re-executes the Workflow code from the beginning, matching each command against the recorded history. random.random() produces different values on each call because it uses the system's entropy source, which differs between the original execution and the replay. If the random value was used to construct a cache key that influenced which Activity was called, the replay produces a different Activity call than the original, and Temporal raises NonDeterminismError because the command sequence does not match. The fix is to use workflow.random() which is seeded deterministically from the Workflow's run ID, producing the same value on every replay. Similarly, datetime.now() must be replaced with workflow.now(), and uuid.uuid4() with workflow.uuid4(). Direct HTTP calls, database queries, and file I/O are also prohibited inside Workflows and must be wrapped in Activities. The error is not about CPU time, import blocking, or payload size.

Why

Temporal recommends Continue-As-New at approximately 50,000 events, when replay time reaches about 10 seconds and starts affecting task pickup latency. At 26,000 events with 5-second replay, the agent is approaching but has not yet reached the threshold. The pattern is to run for a configured number of iterations, then call workflow.continue_as_new carrying only essential summarized state: counters like total checks performed, the latest health check result, configuration values, and any state needed for the next batch. The full historical details are archived and remain queryable but are no longer loaded on every replay. Continuing after every check would lose the ability to track trends across checks and add unnecessary overhead from starting new Workflow executions. A rigid 24-hour boundary ignores the actual event count which is the metric that affects replay performance. Waiting for storage quota exhaustion misses the replay performance degradation that occurs well before storage limits are reached. The state passed to continue_as_new must be minimal and should never include the full conversation history of a long-running agent.

Why

The three planes evolve at different cadences. Intelligence changes weekly through prompt tuning, model upgrades, and tool schema changes. Policy changes monthly through new agent onboarding and permission adjustments. Runtime changes quarterly through infrastructure migrations and scaling events. If the planes are coupled into a single deployable unit, every change to any plane requires a synchronized release. The quarterly Runtime cadence becomes the bottleneck: a prompt tweak that takes 5 minutes to implement cannot ship until the next quarterly Runtime release. This delays the most frequent improvements by weeks or months. Decoupled planes allow independent upgrades: swapping a model requires changing one config string in the Routing plane, tightening a permission requires adding one rule in the Policy plane, and moving from Docker to Kubernetes requires only Runtime changes. No single framework spans all three planes, and production systems compose tools from each plane precisely to maintain independence. The problem is not about memory pressure, circular dependencies, or horizontal scaling constraints.