Idempotency Keys for Agent Side Effects

Before starting

List the actions in the agent run that change another system: creating a resource, sending a message, taking a payment, starting a job, or deleting data. For each one, identify the point at which the intended operation becomes stable and determine the maximum period during which any part of your system can retry it.

You also need durable storage shared by every worker that can execute the action. Process memory is insufficient because a retry may run after a crash, restart, or handoff to another worker. If the run itself must survive those events, settle the broader durable-execution design before adding the per-action protection below.

Steps

1. Put the idempotency boundary around each side effect.

   Protect the smallest operation whose external effect must happen no more than once. One agent run might contain several such operations, and each needs its own key. A key for the whole run is too coarse if later actions must be retried independently; a key for each HTTP attempt is too fine because it treats retries as new work.

   The failure to design for is an ambiguous outcome. The agent sends a request, the downstream system commits the change, and the response is lost before the agent receives it. From the agent’s view, the attempt failed. Retrying without idempotency can repeat the already-successful side effect.

   Do not rely on a durable workflow step alone to close this gap. Cloudflare’s workflow rules, accessed 2026-08-26 say that side effects outside a durable step may be repeated and that deterministic step names act as cache keys. A durable step prevents unnecessary workflow re-execution after its result has been recorded; the downstream idempotency key covers the earlier interval in which the downstream change succeeded but the step result was not recorded.

2. Create one key for the intended operation.

   Generate the key when the system accepts the intent, before the first transport attempt, and save it with the durable run state. The idempotency key must identify the intended operation rather than the transport attempt, so every retry reuses the same key.

   A useful identity has a stable operation identifier and a scope that prevents unrelated tenants or action types from colliding. For example, the operation could be “create asset for approved task 42,” while attempt numbers, worker IDs, timestamps, and retry counts remain transport metadata. Those values must not change the key.

   The IETF Idempotency-Key Internet-Draft published 2025-10-15 describes the key as a client-generated value through which a resource recognizes retries of the same request. It requires a key not to be reused with a different payload and recommends a UUID or similarly random identifier. The document is an expired Internet-Draft, not a published RFC, but its separation of one operation from its retries is the right implementation model.

3. Bind the key to the operation’s immutable input.

   Store a canonical fingerprint of the action type and the fields that define its effect alongside the key. On every call, recompute that fingerprint. If the same key arrives with different effect-bearing input, reject it instead of returning an unrelated cached result or executing changed work.

   This check catches an agent that preserves a key while revising an amount, recipient, asset, or destination. If the user’s intent changes, create a new operation and a new key. Do not silently update the stored input under the old key: that erases the distinction between retrying an operation and requesting another operation.

   Decide deliberately which fields are outside the fingerprint. Attempt count and tracing data do not change the intended effect; a recipient or requested mutation does. The cost is schema work: every side-effecting action needs a reviewed definition of its immutable input. The alternative is a key whose meaning changes during a run, which cannot provide a trustworthy replay result.

4. Reserve the key before calling the downstream system.

   Use an atomic insert or compare-and-set to create a record such as pending for the scoped key and fingerprint. Only the caller that creates that record may begin the downstream call. A concurrent caller that finds pending must not issue the same side effect again; it should return an in-progress result or wait through a bounded mechanism.

   On success, replace pending with completed and store the response needed by the agent: the downstream resource identifier, status, and any stable output required by later steps. When a retry finds completed, return that stored result without calling the downstream service.

   If the downstream API accepts an idempotency key, pass the same operation key on every attempt. This is the important boundary: if the downstream effect commits but your worker dies before changing pending to completed, its idempotency layer can recognize the retry and return the original outcome.

   If the downstream service has no idempotent operation or lookup that can resolve the ambiguous outcome, a local result table does not remove that crash window. The external commit and local completion record are two separate writes. In that condition, do not describe the action as exactly-once; require human reconciliation or redesign the downstream operation before allowing unattended retries.

5. Cache completed results for the entire retry window.

   Set the retention period from the longest retry path, not from the normal request timeout. Include delayed queue deliveries, workflow recovery, scheduled retries, and operator-triggered replay. A result cache for completed keys needs a retention period at least as long as the maximum retry window. Otherwise, a late but permitted retry can arrive after eviction and be mistaken for a new operation.

   Stripe’s idempotent-request documentation, accessed 2026-08-26 provides a concrete implementation: it saves the first request’s status code and body, returns that result for later requests with the same key, compares parameters to reject changed input, and allows keys to be removed once they are at least 24 hours old. That 24-hour value describes Stripe’s stated policy; it is not a default for your system. Your retention must match your own maximum retry window and any shorter downstream expiry must constrain that window.

   Longer retention costs storage and keeps operation metadata available longer. Shorter retention costs correctness. If resources created by the action have their own lifetime, design those expiry semantics separately; expiring a resource does not make reuse of its old operation key safe.

6. Make retry policy consume the stored key, never regenerate it.

   Put the key in the durable action record and have the retry worker load it from there. Do not ask the model to reproduce it from context, derive it from the current time, or generate another key after a timeout. Context can be shortened or reconstructed during a long run; the operation record remains authoritative.

   Retry only under the downstream service’s documented rules. Backoff and admission control still matter because idempotency prevents duplicate effects, not excess traffic. Coordinate them with the system’s provider rate limits and concurrency caps. A retry with the correct key can still consume a request slot or meet an in-progress operation.

7. Test ambiguous and concurrent outcomes.

   Run the action against a controlled downstream stub and cover four cases: two simultaneous requests with one key; a committed side effect followed by a lost response; a retry with the same key and changed input; and a retry at the retention boundary. Count downstream commits, not merely successful agent responses.

   The simultaneous case must produce one downstream execution. The lost-response retry must reuse the key and recover the original result without a second commit. Changed input must fail before execution. A retry inside the declared window must find the completed record; one outside it is no longer protected and should be handled according to the expiry policy you documented.

Expected result

Each side-effecting action has one durable operation record, one stable idempotency key reused by every attempt, an immutable-input check, and a completed-result entry retained through the maximum retry window. When a response is lost after the downstream commit, the retry returns the original operation’s result rather than repeating its side effect.