Set Retry Budgets for Agent Tool Calls

Prerequisites

Before changing retry logic, you need:

• A tool catalog that distinguishes read-only operations from operations that change external state.
• A stable run identifier that is passed through agents, sub-agents, queues, and tool wrappers.
• Structured tool errors. At minimum, the runtime must be able to distinguish a transient failure from a permanent failure and an unknown outcome.
• An idempotency contract for every side-effecting operation you intend to retry.
• A run-level deadline. If you do not have one, set that first using the same request or workflow context described in cancellation and timeout budgets for agent runs.

Do not start by adding a retry library around each client. That creates several independent loops without establishing which layer owns the total cost.

Budget the whole retry tree

1. Find every place that can repeat an operation

   Trace a failed tool call from the model to the downstream service. Include the agent loop, tool wrapper, SDK, HTTP client, queue consumer, workflow engine, sub-agent, and service client. Record which layers retry and whether they share state.

   This matters because a per-call maximum controls only one loop. It cannot stop retries from multiplying when one retrying layer calls another. In AWS’s analysis of timeouts, retries, backoff, and jitter (accessed 2026-08-26), a five-deep service stack with three attempts at each layer can drive 243 calls into the database when the database begins failing. The arithmetic is the mechanism: each parent attempt starts a fresh child retry allowance.

   An agent adds another possible parent loop. A tool wrapper may stop after its configured maximum and return an error, but the model can select the same tool again, a manager agent can reassign the task, or a resumed job can reconstruct a new local counter. None of those layers sees the cost already spent unless the budget travels with the run.

   Choose one runtime layer to own retries for each operation. Lower clients may still handle protocol details, but they must either disable their own retries or debit the same shared budget. A local maximum remains useful as a narrower guard, but it is not the run budget.

   Central ownership costs engineering work because wrappers and SDK defaults must be audited. It is the wrong place to start only when the operation is already governed by a durable workflow engine that exposes and enforces the same shared counters. In that case, use the workflow engine as the owner rather than building a competing loop.

2. Create one budget record and pass it by reference

   Give each run a budget with three independent limits: total attempts, elapsed time, and side effects. A call may proceed only while all applicable limits allow it.

   Budget	Charge it when	What it limits	Why it cannot replace the others
   Total attempts	Before every tool dispatch, including the first attempt	Aggregate load and repeated work across the retry tree	A small call count can still run past the user’s deadline or repeat a dangerous mutation
   Elapsed time	From the run’s start through calls, backoff, handoffs, and queue waits	End-to-end latency	A short deadline does not bound how many fast calls can hit a failing dependency
   Side effects	Before every dispatch that may change external state, including an attempt with an unknown result	Repeated mutations and exposure to uncertain outcomes	An attempt cap treats a status read and a payment request as equivalent

   Store absolute values where possible: a deadline rather than a fresh duration, and counters spent against run-level maxima rather than counters local to a function. A child agent, queue message, or resumed worker receives the existing budget identity and remaining allowance. It must not construct a new allowance.

   Charge before dispatch, not after failure. Otherwise concurrent calls can all observe spare capacity and exceed it together. An operation that times out after reaching the provider has still spent an attempt. If it may have changed external state, it has also spent side-effect budget even though the caller did not receive confirmation.

   The three limits form an AND gate, not fallback choices. Remaining attempts do not authorize a call after the deadline, and remaining time does not authorize another mutation after the side-effect allowance is gone.

3. Derive the limits from the workflow’s constraints

   Do not copy one maximum across every tool. Set each dimension from the condition it protects.

   Start with elapsed time. Reserve part of the run deadline for returning a response, persisting state, or transferring control. Tool calls and retry delays must fit inside what remains. If the next call cannot complete within the remaining time under its own timeout, do not start it.

   Set the total-attempt allowance from the amount of repeated downstream work the workflow can impose. Count attempts across sibling tools when they contend for the same dependency or when the user experiences them as one task. A search call and a record write may have different local policies, but both debit the run total.

   Set side-effect allowance from the operation’s consequence and permission boundary. A run allowed to query repeatedly does not thereby gain permission to send repeated messages, create repeated orders, or update the same record through alternate tools. Tie this allowance to the workflow’s tool permission model so the retry path cannot perform an action the initial path could not perform.

   Use separate named policies where the conditions differ. An interactive request normally has a tighter elapsed-time constraint than a background job; a high-consequence action may permit no automatic repeat without an idempotency guarantee. The specific values must come from your latency objective, dependency capacity, tool cost, and risk policy. No universal numbers are supplied by the cited guidance, so presenting one set as a default would be unverified.

   A side-effect budget is not a substitute for authorization, transaction design, or compensation. It limits exposure after those controls exist. If an operation is unacceptable even once without human approval, set the permission boundary accordingly instead of treating one automatic attempt as harmless.

4. Classify the failure before considering a retry

   Make retryability a deterministic runtime decision. The model may report context, but it must not decide that an unfamiliar error “looks temporary.”

   AWS Prescriptive Guidance on retry with backoff (accessed 2026-08-26) identifies throttling, temporary network problems, and temporary service unavailability as transient cases. It also says to fail fast on identifiable non-transient errors and warns that retries should operate on idempotent methods because partial updates can otherwise corrupt state.

   Build an allowlist from the contracts of the services you actually call:

   • Classify a documented throttling response as transient when the provider permits retrying it.
   • Classify a documented temporary service-unavailable response as transient.
   • Treat a temporary network failure as transient only after resolving whether the operation is safe to repeat.
   • Classify known validation, authorization, or other non-transient failures as permanent and return them immediately.
   • Classify an unrecognized failure as unknown, not transient. Unknown is a stop condition until the contract is updated.

   Keep classification separate from policy. transient means the cause may clear; it does not mean a retry is authorized. The runtime still has to prove idempotency, check all three budgets, and calculate whether another attempt fits before the deadline.

   This allowlist costs maintenance whenever a provider changes its error contract. The alternative—retrying every exception—turns programming errors, invalid requests, and denied operations into repeated load. If a provider does not document whether an error is transient, do not guess from its message text.

5. Require idempotency before retrying any side effect

   A transient classification answers why the call failed. Idempotency answers whether repeating it can change the result twice. Both conditions are required.

   Retry a side-effecting operation only when either:

   • The operation is idempotent by contract, so repeating the same request has the same effect on system state as making it once; or
   • The caller supplies an idempotency key and the receiver guarantees that calls with the same key represent the same logical operation.

   Generate the key from the logical action, not from the attempt. Every retry, handoff, process restart, and model turn must reuse it. If a new attempt receives a new key, the receiver cannot connect it to the earlier request. Persist the key before the first dispatch so a resumed run can recover it.

   The tool contract should expose the guarantee explicitly. A field such as retrySafety: idempotent, retrySafety: idempotency-key, or retrySafety: never is inspectable; a note buried in a prompt is not. The implementation details belong with idempotency keys for agent actions, including how the receiving system stores and resolves keys.

   A timeout on a mutation creates an unknown outcome: the request may have failed before execution, or the response may have been lost after execution. Without an idempotency guarantee, do not retry even if the timeout itself is transient. Return the unknown outcome for reconciliation or human review.

   Idempotency has concrete costs: the receiver needs a stable operation identity, persisted results or deduplication state, and rules for conflicting reuse. It is the wrong answer when the underlying action cannot be made repeat-safe. In that case, use one attempt followed by status reconciliation, human handling, or an explicitly designed compensating action; a retry budget alone cannot make the mutation safe.

6. Put a single retry gate in front of every repeat

   Implement one function that evaluates the full policy before scheduling another attempt. Its decision should depend on structured state, never on a natural-language instruction to “try again carefully.”

   decideRetry(operation, failure, budget):
       if classify(failure) is not TRANSIENT:
           return STOP_PERMANENT_OR_UNKNOWN
   
       if operation changes state
          and operation is not idempotent
          and operation has no reusable idempotency key:
           return STOP_UNSAFE_TO_REPEAT
   
       if budget.totalAttemptsRemaining is exhausted:
           return STOP_ATTEMPT_BUDGET
   
       if budget.elapsedTimeRemaining cannot contain delay plus call timeout:
           return STOP_TIME_BUDGET
   
       if operation changes state
          and budget.sideEffectsRemaining is exhausted:
           return STOP_SIDE_EFFECT_BUDGET
   
       atomically charge the applicable counters
       return RETRY_WITH_SHARED_CONTEXT

   Treat the initial dispatch as an attempt. Naming the field maxRetries often excludes initial calls and makes nested accounting harder to inspect; a totalAttempts limit states the resource being bounded.

   Every execution path must use the gate: direct model tool calls, sub-agent tools, queue redelivery handlers, workflow resumes, fallback tools that reach the same dependency, and manual retry buttons that continue the same logical run. If an operator intentionally starts a new run, record that as a new decision rather than silently resetting the failed run’s counters.

   Shared accounting adds coordination cost, particularly when siblings run concurrently. The counter update has to be atomic or otherwise serialized. If the platform cannot provide shared accounting, disable parallel retries for that operation; independent local counters do not enforce a total budget.

7. Apply backoff and jitter inside the remaining time

   For an authorized transient retry, delay the next attempt using capped backoff with jitter. Backoff reduces the rate at which a failing dependency receives new work; jitter prevents many callers that failed together from returning on the same schedule. Neither makes a permanent error retryable, and neither makes a mutation idempotent.

   Calculate the delay only after checking the shared budget, then check elapsed time again before sleeping and before dispatch. The sleep, queue delay, and next call timeout all consume the same elapsed-time allowance. If they do not fit, stop immediately instead of waiting merely to discover that the deadline has passed.

   Keep backoff at the retry-owning layer. Stacking SDK backoff under workflow backoff both lengthens the schedule and hides elapsed time from the parent. Where an unavoidable client retries internally, configure it from the remaining run budget and report its actual attempt count to the owner.

   The cost of backoff is user-visible or job-visible latency. It is the wrong response when the failure is permanent, the deadline is already too close, or the dependency has given a stop signal. It also does not repair overload by itself: once the budget is spent, callers must stop rather than continue at the capped delay forever.

8. Return a typed terminal outcome when any budget is exhausted

   Budget exhaustion is a runtime state, not a new prompt for the model. Return a typed failure containing enough information for the caller to decide what happens next:

   {
     "type": "RetryBudgetExhausted",
     "operation": "tool-operation-name",
     "reason": "attempts | elapsed_time | side_effects",
     "failureClass": "transient | permanent | unknown",
     "attemptsUsed": "recorded count",
     "elapsedTimeUsed": "recorded duration",
     "sideEffectAttemptsUsed": "recorded count",
     "lastFailure": "structured provider error",
     "idempotencyReference": "stored logical-operation reference",
     "nextAction": "return | reconcile | escalate"
   }

   Do not return a loose string such as tool failed; try another approach. A model can interpret that as permission to call the same tool, select an alias for it, delegate to another agent, or begin a fresh plan. The orchestrator must recognize the typed terminal outcome and block further calls charged to the exhausted dimension.

   Choose the terminal route before deployment. A low-consequence read may return a typed failure to the application. An unknown mutation outcome needs reconciliation or human review. A repeatedly failing background job may move to a dead-letter queue for agent jobs with the budget state and idempotency reference preserved.

   The OpenAI practical guide to building agents (undated PDF, accessed 2026-08-26) recommends setting limits on agent retries or actions and transferring control to a human after the agent exceeds those failure thresholds. It also identifies sensitive, irreversible, or high-stakes actions as triggers for human oversight.

   Human escalation costs attention and response time, so do not use it for every ordinary permanent error. It is appropriate when a person can resolve ambiguity, authorize a new action, or reconcile an unknown side effect. If no human queue exists, return the typed failure; do not pretend escalation occurred and do not let the model improvise another retry loop.

9. Test the budget as a run-level invariant

   Test the failure paths before testing recovery rates. Your suite should prove that:

   • A parent agent and child agent cannot each obtain a fresh attempt allowance.
   • Concurrent tool calls cannot spend the same remaining attempt or side-effect slot.
   • Backoff and queue waits consume the original elapsed-time budget.
   • A permanent or unknown error stops without a retry.
   • A transient error on a non-idempotent mutation stops without a retry.
   • Every retry of a key-protected action reuses the original idempotency reference.
   • An ambiguous mutation result spends side-effect budget and routes to reconciliation or escalation.
   • Exhausting any one dimension prevents another applicable dispatch even when the other dimensions have capacity.
   • A typed terminal outcome cannot be converted into another tool call by an agent handoff or workflow resume.

   Record the run identifier, operation, attempt ordinal, failure classification, remaining budget before and after the decision, delay, idempotency reference, and terminal route. Those records let you distinguish a dependency that failed repeatedly from an orchestration layer that accidentally reset its counters.

   Do not tune the limits from successful runs alone. Exercise throttling, temporary unavailability, lost responses, concurrent siblings, process restarts, and exhausted budgets. The purpose is not to prove that retries always recover. It is to prove that failed recovery remains bounded and leaves an inspectable outcome.

Expected result

A completed implementation has one retry owner and one budget shared across the full agent run. Every dispatch charges total attempts; time spent calling, waiting, and handing off consumes one deadline; every possible mutation charges side-effect budget. Only classified transient failures reach the retry gate, and side effects also require idempotency or a reusable idempotency key. Exhaustion produces a typed failure, reconciliation path, dead-letter route, or human escalation. The model has no path that silently resets the budget or starts another retry loop.