Durable Execution for Long-Running Agent Workflows

What durable execution is

A durable workflow engine persists the position of a run so that a crash resumes from the last completed step instead of restarting the whole sequence. The orchestration logic is ordinary code, but the engine records every step it completes, and on restart it rebuilds the run’s state from that record and continues from where the process died.

For agent workloads, the run in question is usually a chain of model calls, tool invocations, and waits — the kind of sequence where step 14 of 20 dying means throwing away every token spent on steps 1 through 13.

How the position is persisted

The mechanism is an event history plus replay. The engine appends each completed step’s result to a durable log. When a worker picks the run back up, it re-executes the workflow function from the top, but every call that already has a recorded result returns that recorded value immediately instead of doing the work again. Execution fast-forwards through the completed prefix and only does real work once it reaches the first step with no recorded result. Temporal’s workflow documentation describes this execution model and the guarantees it provides.

This is why the distinction between workflow code and activity code exists. Workflow code is the orchestration — the branching, the sequencing, the loop over a list of tool calls. Activity code is the part that touches the outside world. Only activity results go in the history.

The constraint you pay for it

Durability is bought with a constraint: workflow code must be deterministic and replayable, which rules out reading the clock, generating random values, or calling the network outside an activity. Replay only reconstructs the correct state if re-running the function produces the same sequence of decisions it produced the first time. A Date.now() in workflow code returns a different value on replay, a branch taken on that value goes the other way, and the engine’s reconstructed state no longer matches what actually happened.

The practical consequences are specific:

• Time comes from the engine’s own clock API, not the language’s. The engine records the original value and returns it on replay.
• Randomness, including anything that seeds an ID, comes from the engine’s deterministic random source or from an activity.
• Network and disk — every model call, every database write, every HTTP request — go in activities. This is the rule that bites hardest on agent code, because a naive agent loop calls the model directly inside the orchestration function.
• Mutable global state and unordered iteration are also hazards, for the same reason: they make the second run diverge from the first.

The cost is not just the initial rewrite. It is a standing tax on every change, because editing a workflow function that has runs in flight can break the replay of those runs — most engines require versioning gates around changes to already-deployed workflow logic. Teams weighing this against a plain process should read it alongside the trade-offs in serverless functions versus long-running hosts for agent workloads, since the hosting model and the durability model constrain each other.

When it is worth the constraint

The threshold where it becomes worth the constraint is when a single run is long enough or expensive enough that losing it midway is unacceptable — not when the code merely has several steps. Step count is the wrong trigger. A five-step sequence that completes in 400ms and costs a fraction of a cent can simply be retried whole; the determinism tax buys nothing there.

What moves a workload over the line is the cost of a lost run:

• Duration. A run that takes 40 minutes cannot be restarted from zero on every deploy, and deploys are frequent.
• Money. A run that burns a large volume of frontier-model tokens is expensive to redo, and that cost is only visible if you are already tracking agent cost and latency attribution per run.
• Side effects. A run that has already sent an email, filed a ticket, or moved money cannot be restarted from zero without either duplicating those effects or building the same idempotency machinery the workflow engine would have given you.
• Human waits. A run that pauses for an approval, sometimes for days, has no process to keep alive. This is the case where the alternative is not a queue but a bespoke state machine in a database.

Agent runs that call out to sandboxed tools sit in this category more often than they look, because the network egress controls on those sandboxes add latency and failure modes that lengthen the run.

Retries, timeouts, and heartbeats are per activity

Retry policy, timeout, and heartbeat are configured per activity rather than globally, because a model call and a database write fail in entirely different ways. A single global policy has to be wrong for one of them.

The shapes differ concretely:

• A model call may run for tens of seconds, is charged per attempt, and fails with provider 429s and 5xxs that clear on their own. It wants a long start-to-close timeout, exponential backoff, and a retry budget that accounts for the fact that each attempt costs money. It is also the activity most likely to hit provider rate limits and concurrency caps, which is a queueing problem the retry policy alone will not fix.
• A database write returns in milliseconds, and a failure that has not cleared in a couple of seconds usually will not clear on the next attempt either. It wants a short timeout and few retries.
• A long-running tool or batch job wants a heartbeat: the activity reports progress periodically, and the engine treats a missed heartbeat as a failure without waiting out a timeout sized for the whole job. Without one, you choose between a timeout so long that a dead worker goes unnoticed for an hour and one so short that healthy long jobs get killed.

Temporal’s retry policy reference documents the parameters — initial interval, backoff coefficient, maximum interval, maximum attempts, and the non-retryable error types list. That last one matters most in practice: a 400 from a model provider for a malformed request will never succeed, and retrying it burns the budget that a genuinely transient failure needed.

Retries also interact badly with load. Marc Brooker’s analysis of retries and backoff makes the point that retries amplify traffic exactly when a dependency is already struggling, and that backoff alone does not fix it — a client-side budget or circuit breaker is what bounds the amplification. Per-activity configuration is what lets you set that budget differently for a rate-limited model endpoint than for an internal service, which matters when a routing layer is already shifting traffic between models on failure.

What a plain queue already covers

A plain queue with idempotent handlers covers a large share of what teams reach for a workflow engine to do. If the unit of work is a single step, or a short chain where each step can be enqueued by the previous one, then at-least-once delivery plus a handler that is safe to run twice gives you crash recovery without any determinism constraint on your code.

The pieces that gets you: the broker holds the message until it is acknowledged, so a crashed consumer means redelivery rather than loss; idempotency keys mean redelivery does not duplicate the side effect; a dead-letter queue catches what never succeeds. That is durable in the sense most teams mean.

What it does not give you is the state between steps. There is no place to hold “we are on iteration 7 of the agent loop, with these three tool results accumulated” other than a row you maintain yourself — and once you are maintaining that row, adding compensation logic and a timer table, you have started writing a workflow engine with none of the testing. The honest boundary is roughly: chains short enough that a per-message idempotency key is the whole state, stay on the queue.

What to check next

• Your engine’s versioning or patching API — the mechanism for changing workflow code without breaking in-flight runs. This is the operational cost that determines whether durable execution stays comfortable after month one.
• Replay testing: most engines ship a test harness that replays a recorded history against your current workflow code and fails if the code has diverged. Wire it into CI. It composes with the broader problem of evaluating non-deterministic systems in CI, though replay testing itself is fully deterministic by construction.
• Signals, queries, and timers — the APIs for sending input into a running workflow, reading its state without disturbing it, and sleeping durably. The durable sleep is what makes a multi-day human approval step cheap.
• Idempotency semantics of your activities, since activity execution is at-least-once. An activity that charges a card must carry an idempotency key regardless of how the retry policy is tuned.
• Per-run history size limits. Engines cap the event history, and a long agent loop can hit that cap; the standard answer is continue-as-new, which starts a fresh run carrying forward only the state you choose.