Optimistic Concurrency for Shared Agent State

Prerequisites

You need a state store that returns a version with each read and can reject a write when that version is no longer current. The version may be an opaque entity tag, a revision number, or another token maintained by the store. It must change whenever the protected state changes.

Define the state transition separately from external work. Given a state snapshot and an intended event, it should produce a candidate next state without sending messages, charging a card, creating a remote resource, or performing another effect that cannot simply be replayed.

Decide which data must change as one concurrency unit. If two fields must agree for the state to be valid, protect them with the same version or update them in one transaction supported by the store. A version attached to only half of an invariant cannot protect the whole invariant.

Make stale writes fail visibly

1. Choose the state boundary before adding retries.

   Protect the smallest record or transactional group that contains the decision being made. For a task claimed by one worker, that might be the task record containing its status, owner, lease information, and result reference. For an agent run, it might be the run record containing the current step, accumulated outputs, and terminal status.

   A boundary that is too small permits invalid combinations. Updating status and owner independently, for example, can leave a claimed task without the worker that claimed it. A boundary that is too large makes unrelated work conflict: two agents changing independent branches of one large document will reject each other even when both changes could have been kept.

   Split state only when the pieces have independent invariants. Do not split it merely to reduce the number of conflicts. If several records must move together, use a transaction whose scope actually covers them or represent the operation as a state machine in one protected record.

2. Return the version as part of every state read.

   Treat the state and its version as one snapshot:

   type Snapshot<T> = {
     value: T
     version: string
   }
   
   const snapshot = await stateStore.read(runId)

   Do not fetch the version in a separate request after reading the value. Another writer could change the state between those requests, leaving the agent with old data and a new token. The read contract should make it difficult to use one without the other.

   For an HTTP resource, use the entity tag returned with the representation. RFC 9110’s conditional-request rules, published in June 2022, define If-Match for making a method conditional on the current representation having the supplied entity tag. The specification requires strong comparison for If-Match because the condition is intended to detect any change to the representation data. It describes conditional state-changing requests as a way to prevent one client from overwriting work performed by another client in parallel.

   Keep an entity tag opaque. The agent needs to return it, not parse meaning from it. If you implement a numeric revision yourself, increment it in the same atomic operation that writes the new state; reading a counter and updating it separately recreates the race you are trying to remove.

3. Compute a candidate transition from that exact snapshot.

   Pass the snapshot value, the intended event, and any already-recorded tool result into a transition function. The output is a candidate, not yet committed state:

   function applyEvent(state: RunState, event: RunEvent): RunState {
     if (state.status !== "waiting_for_tool") {
       throw new InvalidTransition(state.status, event.type)
     }
   
     return {
       ...state,
       status: "ready",
       completedTools: [...state.completedTools, event.toolResultId]
     }
   }
   
   const candidate = applyEvent(snapshot.value, event)

   Check transition preconditions here as well as at the storage boundary. The storage condition answers, “Has this snapshot changed?” The transition function answers, “Is this event valid for the snapshot I read?” You need both. A current version does not make an invalid state-machine transition valid.

   Prefer expressing the intended change as an event such as tool_result_recorded or task_claimed, rather than retaining only a fully materialized replacement document. An event gives the conflict handler enough information to recompute against a newer snapshot. A replacement document only preserves the stale answer.

4. Require the read version on the write.

   Send the candidate state and the exact version returned by the read:

   const committed = await stateStore.replace(runId, candidate, {
     ifVersion: snapshot.version
   })

   The condition must be checked atomically with the write. Do not implement it as “read current version, compare in application code, then write.” A competing writer can commit after the comparison and before your write.

   With HTTP, send the entity tag in If-Match. If the condition is false, the origin server does not apply the requested change and can return 412 Precondition Failed. Make that outcome a distinct application error such as VersionConflict; do not flatten it into a generic network or storage failure.

   Azure Cosmos DB’s optimistic-concurrency documentation, last updated April 27, 2026, provides a concrete implementation: every item has a server-maintained _etag, a client supplies that value through if-match, and an outdated value causes an HTTP 412 response. The document says _etag changes whenever the item is updated and that conflicting writes in a stored procedure cause the transaction to roll back.

   If your store exposes a revision column instead, use the equivalent conditional update:

   UPDATE agent_runs
   SET state = :candidate,
       version = version + 1
   WHERE id = :run_id
     AND version = :version_read;

   Zero updated rows means the precondition failed. It does not mean the candidate can be written unconditionally on a second attempt.

5. On conflict, discard the candidate, re-read, and recompute.

   A conflict proves that the assumptions behind the candidate may no longer hold. Throw away both the stale candidate and its version. Read the new snapshot, apply the original event to that state, and attempt another conditional write:

   async function commitEvent(runId: string, event: RunEvent) {
     for (;;) {
       const snapshot = await stateStore.read<RunState>(runId)
       const candidate = applyEvent(snapshot.value, event)
   
       try {
         return await stateStore.replace(runId, candidate, {
           ifVersion: snapshot.version
         })
       } catch (error) {
         if (!(error instanceof VersionConflict)) throw error
         // Loop back to read and recompute. Never reuse candidate.
       }
     }
   }

   Suppose agents A and B both read revision 7. A records tool result A and commits revision 8. B’s write, still conditioned on revision 7, must fail. B then reads revision 8 and recomputes. The transition may append result B, decide result B is now redundant, or reject the event because A moved the run to a terminal state. Only the transition logic can make that decision from the current state.

   Retrying B’s revision-7 replacement with exponential backoff only delays the same lost-update bug. If the store keeps enforcing the condition, every attempt with revision 7 fails. If the retry path drops or replaces the condition without re-reading, B can erase revision 8. Backoff changes timing; it does not make stale input current.

6. Add jittered backoff only after each re-read and recomputation.

   Under contention, many losing writers may immediately re-read, recompute, and collide again. Delay those fresh attempts to spread the load, but preserve the order: conflict, discard, delay, re-read, recompute, conditionally write. Delaying before the re-read is acceptable; sleeping and then resending the old candidate is not.

   AWS’s exponential-backoff and jitter analysis, published in March 2015 and updated in May 2023, models optimistic concurrency with competing clients. In its simulation, completion required one winning client per round, completion time grew linearly with the number of contenders, and total client work grew with the square of that number. Plain capped exponential backoff still produced clusters of calls. Adding jitter spread attempts and, with 100 simulated clients, reduced the call count by more than half compared with the un-jittered case. Those are simulation results, not a benchmark for your store.

   Use the store or client library’s retry facilities only if they preserve the required semantics. A library retry that repeats the identical conditional request can be useful for a transport failure whose result is unknown, but it cannot resolve a confirmed version conflict. Your application must receive that conflict so it can recompute.

   Bound the retry loop according to the operation’s latency budget. When the bound is reached, return a contention result or place the event back into a controlled work queue. An unbounded loop can keep a run occupied while other writers continue to advance the same record. There is no supplied universal retry count or delay for this case; choose and measure them in the system that owns the workload.

7. Make duplicate events safe during recomputation.

   A response can be lost after a write commits, leaving the caller unsure whether the event was applied. Put a stable event or command identifier in the state transition. Before applying an event, check whether that identifier has already been recorded. If it has, return the recorded outcome instead of applying the transition twice.

   This is separate from the state version. The version detects that something changed since the read; the event identifier establishes whether this particular intent already took effect. You need both when callers may retry after ambiguous failures.

   Keep the recorded identifier and its resulting state change in the same conditional write. Recording it later creates a gap in which the state can change successfully but a retry still appears new. For operations that cross the state-store boundary, use idempotency keys for side-effecting agent actions rather than assuming the state version protects a remote service.

8. Keep external side effects outside automatically replayed state transactions.

   The transition loop may run more than once. Therefore its replayable section must not send an email, publish a message, charge a payment method, create a repository, or call another service merely because the candidate state says to do so. A failed conditional write would replay that call even though the state change did not commit.

   Record intent first. One workable sequence is:

   1. Conditionally move the protected state from ready to effect_pending, including an operation identifier and the exact requested parameters.
   2. A worker claims that pending operation using another conditional transition.
   3. The worker performs the external call with the operation identifier as an idempotency key when the destination supports one.
   4. The worker conditionally records the returned result and moves the state to effect_succeeded.
   5. A reconciler examines operations left pending or with an unknown outcome and checks the destination before deciding whether to retry, complete, or compensate.

   This sequence does not make the state store and remote service one transaction. It makes the incomplete interval explicit and recoverable. Keep external side effects outside an automatically replayed state transaction unless those effects have their own idempotency and reconciliation controls.

   If the destination cannot deduplicate requests, an automatic retry after an unknown result can repeat the effect. The correct response is not to hide that uncertainty in a generic retry loop. Persist the unknown state and provide a destination-specific check or a compensating action for the agent side effect. A compensating action is a later operation with its own failure modes, not a rollback of the original remote action.

9. Treat contention separately from provider throttling.

   A version conflict means another valid state transition won. A rate-limit response means a provider declined an attempt under its current limits. Both may lead to delayed work, but only the first requires re-reading shared state before deciding what the next write should be.

   Keep separate error types, counters, and retry paths. Otherwise a generic retry layer can mistake a version conflict for a transient request failure and resend stale state. Conversely, recomputing state cannot remove a provider’s concurrency cap. Handle those delays through the controls described for agent execution under provider rate limits and concurrency caps.

10. Test the collision, replay, and side-effect boundaries.

    Use a barrier in an integration test so two workers read the same version before either can write. Release both writes and assert that exactly one conditional write succeeds. The loser must read the winner’s state and run the transition again; assert that the final state contains the result required by both valid events or explicitly rejects the event that became invalid.

    Add a test in which the store commits a write but the caller receives no success response. Retry the same event identifier and assert that the transition reports the recorded result without applying it again.

    For the side-effect path, stop execution after the external service accepts the operation but before the state records success. Start the worker again. The test should show that the idempotency or reconciliation control finds the prior outcome rather than blindly issuing the operation again.

    Finally, record version conflicts separately from transport errors, invalid transitions, rate limits, and exhausted retries. A conditional rejection is expected coordination, but a sustained concentration on one record is still operational evidence that the chosen state boundary or work partition deserves review.

Expected result

Every state write is tied to the exact snapshot from which it was computed. A concurrent winner makes a stale write fail without changing stored state. The losing agent reads the new version, recomputes the transition, and submits a new conditional write. Automatically replayed code performs no uncontrolled external effect; uncertain effects remain identifiable and can be deduplicated, reconciled, or compensated.