Rotate Secrets Without Breaking Autonomous Agents

Prerequisites

Use this procedure for planned rotation only. You need a credential issuer that can keep two credentials valid at once, a secret store with versions or aliases, and logs that distinguish credential generations without recording their values. Secret rotation needs an overlap window where old and new credentials both work; otherwise an in-flight agent run can fail between two steps even though deployment health looks normal.

If the provider permits only one active credential, do not attempt a live cutover. Stop new runs, drain every queued and active run that can use the credential, replace it, and then resume. If the old credential may be compromised, containment can take priority over continuity: revoke it and treat interrupted runs as recovery work.

The OWASP Secrets Management Cheat Sheet, retrieved 2026-08-26, separates creation, rotation, revocation, and expiration and calls for auditing when a secret is used, updated, expired, or reused after expiry. Your rotation workflow needs to represent those stages separately rather than treating credential creation as completion.

Rotation procedure

1. Define the rotation boundary and overlap window.

   Name the credential, its stable secret reference, every agent deployment allowed to resolve it, and the provider operation it authorizes. Include scheduled workers, retry queues, paused runs, and delegated execution. The relevant inventory follows the actual agent deployment topology, not merely the repository that starts runs.

   Set the overlap window from observed execution behavior: it must cover the longest permitted run that could have resolved the old credential before cutover, plus any allowed queue and retry delay. A guessed five-minute window is not safe merely because most runs finish sooner. If a run can pause for approval or resume after a long backoff, either include that interval or prohibit such runs from crossing a rotation.

   Record the planned cutover time and the earliest possible revocation time. These are decision gates, not promises: old-credential usage can extend the overlap.

   For cryptographic keys, NIST SP 800-57 Part 1 Revision 5, published May 2020, provides the applicable lifecycle model: keys move through defined states, transitions are recorded, and metadata may remain for audit after destruction. It is guidance for cryptographic keying material, not a universal lifetime schedule for every API token. Use its state discipline without pretending it supplies a rotation interval for an unrelated credential.

2. Make the agent resolve the secret at call time.

   Persist a reference such as secret://payments/agent-api/current in run state, never the credential value. Immediately before each authenticated external call, the tool boundary resolves that reference, uses the returned value for the call, and discards the value after use. A resumed run therefore asks the secret store for the current generation instead of replaying whatever was present when the run started.

   Keep the distinction explicit in agent memory and state: the durable record may contain the secret reference, provider name, intended scope, and non-secret generation identifier. It must not contain the token, password, private key, request header, or a serialized tool argument holding any of them.

   Call-time resolution costs a secret-store lookup and makes that store part of the call path. If that dependency is unavailable, the agent should fail the affected call rather than fall back to a credential copied into run state. Process-local caching is the wrong answer when its lifetime can exceed the overlap window or when there is no reliable invalidation path.

3. Add generation-aware evidence before rotating.

   Emit one record for each secret resolution and authenticated provider call. It should contain the run ID, tool or operation, stable secret reference, secret-store version ID, deployment identity, timestamp, and outcome. The version ID must be metadata assigned by the secret system, not the secret value or a reversible encoding of it.

   Correlate caller records with the provider’s authentication or audit records where the provider exposes them. Caller logs show which generation the agent intended to use; provider records show which credential actually reached the service. Neither alone proves the full path.

   If agent operations already produce OpenTelemetry spans, attach the non-secret rotation fields to the relevant operation using the convention revision your system has pinned. The supplied OpenTelemetry Generative AI semantic-conventions page stated on retrieval, 2026-08-26, that its content had moved and was no longer maintained there. Treat that URL as a pointer, not proof that a particular attribute name is current. Define private field names when necessary and never place a credential value in a span, event, prompt, response, or error message.

4. Issue and stage the new credential.

   Create a second credential with the scope required by the same operation. Store it as a new version beneath the existing stable reference, but do not move the current alias yet. Keep the old version active at the provider.

   Test the new version directly through the same resolver and tool path used by production agents. A successful secret-store write proves only that a value exists. It does not prove that the provider accepts it, that its scope covers the required operation, or that deployed workers can resolve it.

   If the credential was issued inside an agent session, follow the same placement boundary used for newly issued agent credentials: the issuing response should be handed to the secret store, while durable run state retains only the reference and version metadata.

5. Start overlap and move the reference.

   Confirm that both credential generations work, then atomically point the stable reference at the new version. Record the cutover timestamp and new version ID. New call-time resolutions should now return the new credential, while calls already sent with the old credential remain valid during the overlap.

   Do not rewrite queued run records, prompts, or checkpoints. If that appears necessary, the system has persisted secret values or version-pinned credentials in the wrong place. Fix that path before completing the rotation.

   Do not revoke the old credential immediately after the first successful new call. Autonomous runs can be on different steps, retry schedules, or deployments. The overlap exists specifically so those in-flight runs do not fail mid-step while the reference changes.

6. Wait for evidence that old use has stopped.

   Watch resolutions and provider calls by generation. You need both positive evidence that the new credential is being used and negative evidence that the old credential has had no use across the required observation interval.

   Start that interval after the last recorded old-generation use, not merely after cutover. It must be at least the previously defined maximum run, queue, and retry span. Any later old use resets the interval and identifies work to investigate: a cached value, a deployment resolving only at startup, a paused run with serialized credentials, or an unlisted consumer.

   This is the completion test that matters. Confirmation that the new credential exists, resolves, or succeeds does not show that the old one is unused. Rotation completion requires evidence of old-credential non-use before revocation.

7. Revoke the old credential and test both outcomes.

   Once the evidence window passes with no old use, revoke the old credential at the provider. Mark its secret-store version revoked or archived so the stable reference cannot return it. Preserve only the non-secret audit metadata required to explain when it was issued, selected, last used, and revoked.

   Run one call through the stable reference and confirm that it succeeds with the new generation. Separately confirm that the old credential is rejected, using a controlled authentication check that performs no business mutation. Then monitor for attempts to resolve or use the old version. Such an attempt after revocation is a defect even when the agent retries successfully with the new credential.

Expected result

Rotation is done when the stable reference resolves only to the new credential, production calls succeed with that generation, the old credential has shown no use for the full evidence window, the provider rejects it after revocation, and no secret value remains in durable agent state, telemetry, prompts, or error records.