Set timeout budgets for AI agent runs

Prerequisites

Before changing the loop, identify its model-call and tool-call boundaries, the component that notices a client disconnect, and the component that owns the run’s total execution limit. Each boundary needs a way to receive a cancellation signal or deadline. You also need a run record that can be updated after the caller has gone away; otherwise cleanup and the terminal outcome have nowhere reliable to land.

Steps

1. Define cancellation and timeout as separate terminal reasons.

   A client disconnect and an execution timeout are different signals. The disconnect says the caller stopped listening; the timeout says the operation exceeded its budget. Do not collapse both into a generic failure. The distinction affects whether work should stop, what gets reported, and whether another component should retry it.

   At the request boundary, translate the connection ending into a caller-cancellation signal. Keep the run deadline separate. This follows the request and connection boundary described by HTTP semantics in RFC 9110 (June 2022) without pretending that a broken connection proves the operation itself timed out.

   Decide the contract before implementing it. For a synchronous run whose only purpose is answering the connected caller, a disconnect should normally request cancellation. For an accepted background job, the caller may merely detach while the job continues. In that second case, label the event as detachment rather than cancellation; otherwise operators will read an intentional continuation as a propagation failure.

   Use distinct terminal values such as caller_cancelled, deadline_exceeded, completed, and failed. Preserve the first terminal cause. A cleanup request that later times out should not overwrite the fact that the caller cancelled first.

2. Create one absolute deadline when the run is admitted.

   Represent the total budget as an absolute deadline and pass it with the run context. Before starting any model or tool operation, calculate the remaining budget from that deadline. An absolute value prevents each layer from accidentally starting a fresh full-duration timeout.

   Reject or shorten work that cannot fit inside the remaining budget. Starting it with the original allowance defeats the total limit. A host-level execution cap can remain as a final containment measure, but it is the wrong primary control: if the host kills the process, the loop may lose its chance to record the outcome or perform cleanup.

   Host choice still matters. A request-bound function and a durable workflow have different failure boundaries, as discussed in serverless functions versus long-running hosts. If a run is implemented as durable steps, Cloudflare’s Agents workflow model (accessed 2026-08-26) is one concrete example of step-based execution. Durability does not remove the need for a run deadline; the deadline must remain part of persisted run state so resumed work observes the same budget.

3. Divide the run deadline into step budgets.

   Reserve part of the total run budget for cleanup and final reporting before assigning time to model and tool work. Then give each step its own deadline, capped by the run deadline minus that reserve. This is the condition that prevents a late tool call from consuming all the time needed to close resources, store the terminal state, and report what happened.

   Express the rule directly:

   step deadline = earlier of the step allowance and the run deadline minus the reserved closing budget

   Recalculate before every step. Do not allocate the whole remainder once and assume later calls will finish quickly. The mechanism is simple: as the run advances, the available window shrinks, while the closing reserve stays unavailable to ordinary work.

   Set different allowances where the operation justifies them, but do not invent precision unsupported by measurements. Start from service limits and observed run traces, then revise the policy with dated evidence. A uniform per-step timeout is acceptable only when the steps have similar cost and interruption behaviour. It is the wrong answer when one call is predictably long or when a side-effecting tool needs a shorter decision window.

4. Propagate both signals through the entire agent loop.

   Check caller cancellation and remaining time before planning, before each model call, before each tool call, and immediately after either returns. Pass the cancellation handle and the step deadline into every API that supports them. Stop scheduling new work as soon as either signal becomes terminal.

   The control flow should have one shape:

   before each operation:
     if caller cancellation is active: finish as caller_cancelled
     if the run deadline is exhausted: finish as deadline_exceeded
     derive a step deadline that preserves the closing reserve
     call the model or tool with cancellation and that deadline

   Cancellation has to propagate through the loop into model calls and tools. If it stops at the HTTP handler, the visible request ends while model inference, polling, subprocesses, or remote tools can continue consuming paid capacity. Merely abandoning the returned promise or future is not propagation; it stops waiting locally but does not tell the underlying operation to stop.

   A framework is useful only if it exposes these boundaries. When comparing an agent framework with a plain tool-calling loop, verify that cancellation reaches the actual provider and tool clients rather than only interrupting the framework’s scheduler.

5. Give non-cancellable tools an explicit containment policy.

   Some interfaces will not accept a cancellation signal. Mark those calls in the tool registry rather than treating them as cancellable. Give them a step deadline, stop polling when the run terminates, and do not schedule dependent steps afterward. Record that the remote operation may still be running.

   Cancellation also cannot retract a side effect that already completed. Check the signal immediately before authorising the effect, and make the terminal record distinguish “cancelled before invocation” from “cancellation requested after invocation.” For consequential tools, the agent tool permission model should decide whether the operation may begin when little budget remains.

   The cost of stricter containment is that useful late results may be discarded. The alternative cost is harder to bound: work can continue after the caller and scheduler believe the run is over. Choose per tool, based on whether an unfinished call is read-only, billable, or externally consequential.

6. Limit retries and fallbacks by the remaining run budget.

   A retry or fallback is another step, not an exception to the deadline. Before starting one, derive a fresh step deadline and preserve the same closing reserve. If it cannot fit, finish with the existing terminal reason instead of beginning work that the run cannot report cleanly.

   Keep cancellation stricter than ordinary failure: caller cancellation should not silently trigger a fallback model. A timeout may permit a fallback only when the remaining budget can accommodate it and the run policy explicitly allows it. Put those conditions in the fallback model policy rather than scattering them across provider-specific error handlers.

7. Record where the budget went and why the run ended.

   Emit a span for the run and child spans for model and tool operations. Use the OpenTelemetry generative-model semantic conventions (accessed 2026-08-26) as the naming baseline for model operations, then add local attributes for the run deadline, step deadline, remaining budget at start, cancellation source, terminal reason, and whether closing work completed.

   Carry one correlation identifier across the request, run, model calls, and tools. The correlation identifier design should let an operator reconstruct whether the client left first, a step exhausted its allowance, or the total deadline expired. Do not record every timeout as an undifferentiated error; that destroys the distinction the control flow depends on.

8. Test each termination path at a real boundary.

   Test a disconnect during a model call, a disconnect during a tool call, a step timeout while the caller remains connected, and exhaustion of the total deadline near the closing reserve. Also test a tool that ignores cancellation. For each case, confirm that no new step starts, cancellable work receives the signal, the correct terminal reason is preserved, and cleanup and reporting use only their reserved budget.

   Test the background-job contract separately. Disconnecting its submission request should detach the caller without cancelling accepted work, while an explicit cancellation command should propagate through the resumed loop. This prevents request lifecycle rules from leaking into durable job semantics.

Expected result

The finished implementation distinguishes caller cancellation from budget exhaustion, carries both through every model and tool boundary, prevents any ordinary step from spending the closing reserve, and records one stable terminal reason. A disconnected synchronous request stops cancellable paid work, while accepted background work continues only when its contract explicitly treats the disconnect as detachment.