Add circuit breakers to AI agent tool calls

Prerequisites

Before configuring a breaker, identify each remote tool call, its timeout, the errors it returns, whether retrying can repeat a side effect, and what the agent should do when the tool is unavailable. The agent must be able to persist or terminate its current step without pretending that a rejected call succeeded. If failed work can wait, define the handoff to a dead-letter queue for repeatedly failing agent jobs before changing admission behavior.

Steps

1. Choose a breaker boundary that matches the failure boundary

   Put the breaker immediately around the client operation that reaches the remote dependency. Use separate breakers when operations or backends can fail independently: an outage in a search tool should not disable object storage, and one regional endpoint should not necessarily block another healthy region.

   Do not create a breaker per agent run. That state disappears before it can protect later runs. At the other extreme, one global breaker for every tool turns a narrow failure into a system-wide stop. Share state across the workers that send load to the same dependency, or accept that each process will admit its own probes and calculate the resulting probe load.

   Finer boundaries isolate failures more accurately but create more state, metrics, and configuration. Combine operations only when they use the same capacity and normally recover together. If the tool client is buried inside an agent framework, put the breaker at the client boundary rather than scattering state transitions through prompts or planning code; that distinction also matters when choosing an agent framework or a plain tool-calling loop.

2. Classify the outcomes that affect the breaker

   Count failures that indicate the dependency is unavailable or too slow: the configured timeout, a failed connection, or a service response that explicitly reports temporary unavailability. Do not count a valid tool response merely because the model cannot use its result. Authentication, malformed arguments, and policy rejection need their own handling because waiting for dependency recovery does not change the request.

   For HTTP tools, preserve the response details. RFC 9110’s HTTP semantics, published June 2022, defines 503 Service Unavailable as temporary inability to handle a request and allows Retry-After to state an HTTP date or a delay in seconds. Treat that value as evidence for the recovery interval, not as permission to replay an unsafe operation automatically.

   Record successes as well as counted failures. A raw failure total without the number of admitted calls cannot distinguish a broken dependency from one bad call in light traffic. Keep client cancellations separate where possible: a run cancelled by its caller says nothing about tool health unless the tool itself exceeded the client’s deadline.

3. Keep retries behind the breaker

   Retries answer whether one request should be tried again; a circuit breaker answers whether new requests should be admitted at all. The agent should ask the breaker for admission before every original attempt and retry. Once the breaker is open, retry code must stop instead of treating the breaker’s rejection as another transient tool error.

   Bound retries by both attempt count and the agent step’s remaining deadline. Apply them at one layer so an SDK, tool adapter, and agent loop do not each multiply attempts. For eligible transient failures, use backoff and jitter. The AWS Builders’ Library guidance on timeouts, retries, backoff, and jitter explains that retries add load to the dependency, capped backoff limits their rate, and jitter prevents clients from lining up for another simultaneous attempt (source retrieved 2026-08-26).

   Retrying is the wrong answer when the operation can create the same external effect twice and the tool offers no idempotency mechanism. It is also wrong after a permanent request error. In either case, fail the step with the original evidence rather than spend the retry budget.

4. Define the closed-to-open rule from recent behavior

   In the closed state, admit calls and maintain a time-bounded record of the classified outcomes. Open only when the chosen threshold is crossed within that window. Microsoft’s circuit-breaker pattern, last updated 2025-03-21, describes this recent-failure threshold and the closed, open, and half-open state machine.

   Choose the window and threshold from observed call volume and failure episodes. Require enough observations that one isolated timeout cannot open a high-volume shared breaker accidentally. A threshold that is too sensitive rejects healthy traffic during brief noise; one that is too tolerant lets many calls occupy agent deadlines while the dependency is plainly failing. There is no supplied universal number, so a value not derived from your traffic remains unverified.

   Emit a state-change event containing the dependency, operation, previous and new states, counted outcome, threshold window, and configuration version. Do not include tool credentials, prompts, or full payloads. Pair aggregate breaker metrics with trace sampling for agent systems so an operator can inspect representative failures without retaining every call.

5. Make the open state a defined agent outcome

   Open means new calls fail immediately without reaching the dependency. Return a typed tool_unavailable result containing the tool identity, breaker state, and earliest automatic probe time. This is an admission decision, not proof that the current request itself would have failed.

   Decide the agent response per tool before release:

   • Use a fallback only when it preserves the step’s meaning and the caller can identify that substitution.
   • Defer durable, still-useful work to a queue; do not queue work whose deadline or business value will expire first.
   • Stop the run when the missing tool is required for correctness.

   The open interval must be long enough for the dependency to recover but short enough to probe recovery without a manual reset. Use a valid Retry-After signal when available; otherwise start from observed recovery time and revise it from incidents. A short interval repeatedly loads a dependency that is still failing. A long interval extends an outage after the dependency has recovered. Keep manual force-open and reset controls for operations, but do not make routine recovery depend on them.

6. Limit half-open probes and close cautiously

   When the open interval ends, move to half-open rather than closed. A half-open state should admit only a small number of probes or a recovering service receives the original load immediately. Enforce that limit atomically across workers sharing the breaker; a per-worker limit multiplied by many workers is not small at the dependency.

   Let ordinary representative calls act as probes only when their side effects are safe. Otherwise use a documented health operation, while recognising that a shallow health check might succeed before the real operation does. Successful probes move the breaker toward closed according to the configured success rule. A counted probe failure returns it to open and restarts the recovery interval.

   Probe limits trade recovery speed for protection. One in-flight probe gives the clearest isolation but can delay confirmation when it is slow. More probes provide evidence sooner but consume more recovering capacity. Select the number from dependency capacity and worker count; no universal probe count is established by the supplied evidence.

7. Test transitions under concurrency

   Exercise the breaker with timeouts, temporary service errors, permanent request errors, an open interval, successful recovery, and a failed half-open probe. Run the test with the same worker concurrency used in production. Verify that open calls do not reach the tool, retry loops stop, the half-open cap holds globally, and only the intended breaker changes state.

   Alert on sustained open state, repeated open–half-open cycling, and probe failure. Dashboard admitted, rejected, successful, failed, and probe calls separately. A falling tool-call count during an open circuit is expected; without the rejection count it can be mistaken for recovery or lost traffic.

Expected result

Done means a failing tool receives no ordinary calls while its breaker is open, retries cannot bypass that decision, and only the configured small probe set reaches it in half-open. Agent runs follow an explicit fallback, deferral, or failure path, while state-change events and metrics show why admission stopped and when automatic recovery was tested.