Set a Fallback Policy for AI Agent Model Failures

Prerequisites

Before adding a fallback, make these parts of the run explicit:

• the tools the agent may call, including their input and result schemas;
• the final output schema or acceptance check;
• the run’s overall deadline and per-attempt timeout;
• the context needed to resume after each completed step;
• a failure envelope that distinguishes transport, safety, tool-call, validation, and context failures.

Without that envelope, the router cannot tell whether another model is a recovery mechanism or an attempt to bypass a valid rejection.

Configure the policy

1. Freeze the run contract before choosing models.

   Write one versioned contract for the whole run. It should name the allowed tools, their schemas, the final output shape, permission boundaries, and the state that must survive between model calls. Apply the same contract to the primary and every fallback.

   Do not give the fallback broader tool access to compensate for weaker tool selection. A model change is not a permission change. If a tool result later enters context, keep the same tool-result validation boundary regardless of which model requested the call.

   The cost is maintenance: every schema or permission change must update the compatibility tests for each eligible model. The wrong shortcut is treating a model identifier as the policy. A name says which model receives the request; it does not prove that the model can complete this run under the same rules.

2. Classify the failure before routing anywhere.

   Route from a typed failure, not from a catch-all exception. A transport failure can justify a fallback because the application did not receive a usable model result. A safety refusal or invalid tool request usually should not be retried on a different model: the first is a policy outcome, while the second is a malformed action that the application should reject or repair through its defined error path.

   Use a decision table in the policy:

   Observed outcome	Fallback eligible?	Required handling
   Transport ended without a usable response	Yes, if budget and contracts permit	Retry or change model under the bounded retry policy
   Safety refusal	Usually no	Return or escalate the refusal according to product policy
   Invalid tool request	Usually no	Reject it; use the normal repair path if one exists
   Final output fails validation	No automatic model switch	Apply the declared output-repair or failure path
   Required context does not fit the fallback	No	Compact from an approved checkpoint or end the run

   This classifier costs implementation work because provider errors must be translated into your own stable categories. It is still cheaper than debugging a router that treats every non-success as an outage. Do not classify from error text alone when a structured status is available.

3. Put retries and fallback inside one deadline.

   Give the complete run a deadline, then allocate attempts within it. Do not let each model receive a fresh full timeout. That turns a recovery path into an unbounded extension of the original request.

   The AWS guidance on timeouts, retries, backoff, and jitter describes retries as load-amplifying behavior and recommends bounded retries with backoff and jitter. Applied here, the policy should state the maximum primary attempts, fallback attempts, delay ceiling, and total elapsed-time limit. A defensible starting policy for an interactive run is one primary attempt and at most one fallback attempt; that is an operating choice to test, not a universal performance claim.

   Stop when the remaining time cannot cover another attempt and its result validation. Coordinate this with the agent’s cancellation and timeout budgets. Backoff is the wrong answer when the caller’s deadline has already expired, and immediate repeated attempts are the wrong answer when the failed service is under pressure.

4. Prove tool and output compatibility.

   The fallback must satisfy the same tool and output contracts, or recovery creates a second class of malformed results. Compare more than tool names. Check required arguments, allowed values, result handling, approval requirements, and the final structured output schema.

   Build a conformance suite from representative run states. For every eligible model, verify that valid tool calls pass, invalid calls are rejected, required fields remain required, and the final result passes the same validator. Record the contract version with the run so a later schema change does not make an old trace ambiguous.

   Compatibility testing costs evaluation time and may disqualify a cheaper or more available model. That is the correct outcome when the route requires a tool or output shape it cannot reliably produce. Do not weaken validation only on fallback responses; that hides recovery failures by admitting results the primary route would reject.

5. Check context capacity at the handoff point.

   Calculate the resume payload before selecting the fallback: system instructions, active user input, retained model items, validated tool results, unresolved decisions, and space reserved for the next response. A smaller context window can make a nominal fallback incapable of resuming the original run.

   Define one of three outcomes when the payload is too large:

   • mark that fallback ineligible for this run;
   • compact at a previously tested checkpoint and resume from the compacted state;
   • stop and return a typed context-capacity failure.

   Silent truncation is not recovery. It can remove a permission boundary, completed action, or required evidence while making the new attempt look valid. A compacted route costs tokens and engineering work, and it must preserve the facts the next step needs. Use the same tested rules as the site’s guidance on keeping long-run context useful. Compaction is the wrong answer when the omitted material is itself required evidence.

6. Resume from a committed checkpoint, not from the beginning.

   Store a checkpoint after each accepted model result and each validated tool result. On fallback, resume after the last committed item. Do not repeat completed tool calls merely because a new model took over.

   OpenAI’s current model guidance, checked 2026-08-26, tells applications that manage history manually to preserve and resend prior user inputs and response output items; its programmatic tool guidance also preserves call identifiers and caller linkage. The general policy consequence is clear: a fallback needs the state and identifiers required to continue the same run, not a reconstructed prompt that loses which calls already completed.

   Checkpoint storage adds data and lifecycle work. Replaying the entire run is the wrong answer when tools can create side effects. Carry a stable run identifier, attempt number, and tool-call identifier through the handoff, following the same scheme used for correlation identifiers across model calls and tools.

7. Trace the routing decision as a first-class event.

   Record the requested model, responding model, attempt number, failure class, fallback decision, contract version, checkpoint identifier, and whether context compaction occurred. Keep refusal, invalid-tool, timeout, and transport outcomes separate; otherwise a dashboard can show a high fallback rate without revealing whether the system is recovering outages or retrying policy failures.

   Use the OpenTelemetry generative-model semantic conventions, checked 2026-08-26, as the common vocabulary for model operations and errors, then add application fields for the fallback decision. The supplied OpenTelemetry URL now points readers to the maintained GenAI conventions repository.

   More trace fields increase telemetry volume and may expose sensitive content if prompts or results are recorded. The wrong answer is logging full context by default. The routing policy needs identifiers, classifications, model names, and contract versions; content capture should follow a separate retention decision.

8. Test failures, not only successful failover.

   Run controlled cases for a lost transport response, a safety refusal, an invalid tool request, an output-schema violation, and a resume payload that exceeds the fallback’s context capacity. Assert both positive and negative routes: the transport case may reach the fallback, while the refusal and invalid tool request do not. Assert that an over-capacity fallback is rejected before invocation.

   Also verify that the second attempt retains the original deadline, permissions, checkpoint, tool contract, and output validator. A fallback test that checks only whether some text came back does not test recovery.

   These tests add fixtures and evaluation runs. They are unnecessary only for a system that has no fallback path. Once automatic routing exists, leaving its refusal and malformed-result branches untested makes the exceptional path the least specified part of the agent.

Expected result

The policy is complete when every failed model call receives a typed outcome; only eligible transport failures can consume the bounded fallback attempt; refusals and invalid tool requests remain on their declared failure paths; and every fallback proves contract and context compatibility before invocation. The resulting trace identifies why routing occurred, what state resumed, and which validator accepted or rejected the result.