Version and Release Prompts for AI Agents

Prerequisites

Before releasing a prompt independently of application code, you need four things:

• One assembly path through which every instruction passes before a model request is sent.
• An immutable place to retain released prompt artifacts and their component manifests.
• Run telemetry that can carry prompt and model identities.
• A runtime switch that can select the previous or candidate prompt without rebuilding the service.

If instructions are assembled in several clients, normalize that first. Otherwise the same release label can describe different model inputs, and neither evaluation nor rollback has a precise target.

Steps

1. Set the version boundary at the final assembly point

   A prompt version must identify the full assembled instruction set, not only the editable template, because hidden preambles and retrieved policy change behavior. The version boundary therefore belongs immediately before the model request, after your application has combined every instruction-bearing component.

   Include, in delivery order:

   • the application or framework preamble;
   • the editable task template;
   • policy text selected or retrieved for the run;
   • tool-use instructions and tool descriptions;
   • output-format and stopping instructions;
   • any wrapper text inserted by an SDK, gateway, or orchestration layer.

   Keep user input and ordinary retrieved evidence outside the release identity unless your application treats them as instructions. They still belong in the run record. Retrieved policy is different: if it directs what the agent may do, changing it changes the assembled instruction set and must produce a different fingerprint.

   Capture two identifiers. Use a readable release ID, such as support-agent/2026-08-26.1, for operations, and calculate a content fingerprint from the ordered assembled instructions. The release ID tells an operator what was approved; the fingerprint detects two supposedly identical releases that assembled different content.

   Record a manifest rather than relying on the final text alone. Each entry should identify the component, its revision or content fingerprint, its position, and whether it was static or selected at runtime. Also version the assembler itself. A changed join order, role assignment, or separator can change the model input even when every component file is unchanged.

   A provider-controlled preamble that the provider does not expose cannot be captured as part of your artifact. Record that limitation alongside the provider and model version. In that condition, you can reproduce your application-controlled input but should not describe the whole request as exactly reproducible.

   The cost is storage for immutable artifacts and extra work in the request path. The wrong shortcut is assigning a version to one template file: it leaves policy, wrappers, and assembly order outside the release you think you tested.

2. Publish an immutable prompt release artifact

   Turn the assembled prompt definition into an artifact before running release evaluations. A release should contain:

   • the readable release ID and content fingerprint;
   • the ordered component manifest;
   • the assembler revision;
   • the compatible agent and tool-schema versions;
   • the evaluation-dataset revision and acceptance rules;
   • the preceding production release ID;
   • its lifecycle state: candidate, canary, production, rolled back, or retired.

   Immutability matters operationally: correcting a candidate in place would make earlier evaluation results refer to different content under the same name. Publish a new release ID for every changed instruction set, even for punctuation or a single word. The release note may call the change small; the identity must still change.

   Do not make semantic version labels carry more meaning than your process can enforce. A content fingerprint is the precise identity. A readable label is an index for people. Neither proves that two releases behave compatibly; evaluations supply that evidence.

   Retain the previous production artifact while its successor is being evaluated and canaried. A rollback flag is useless if the old artifact has been overwritten or its retrieved policy can no longer be resolved.

   The cost is registry maintenance and retention. This is the wrong design when the registry stores only mutable pointers such as current or latest; those names are useful for routing, but they are not versions and cannot identify a historical run.

3. Record the prompt and model identities on every run

   Every run should record the prompt version beside the model version so an evaluation can be reproduced. Store both the readable prompt release ID and its assembled-content fingerprint. For the model, distinguish the model requested by your application from the model reported in the response when the provider supplies both.

   At minimum, a run record needs:

   • run ID and start time;
   • prompt release ID and assembled fingerprint;
   • ordered component manifest or a reference to the immutable manifest;
   • requested and returned model identities;
   • model parameters that the application set;
   • tool-schema revisions available to the run;
   • selected policy revisions;
   • rollout flag state and cohort;
   • input references, output, tool calls, and stopping result, subject to your data-handling rules.

   Put these fields on the trace or run envelope, not only in deployment logs. One deployment can serve old and candidate prompts at the same time, so a deployment version does not identify which prompt handled a request.

   Align the model fields with the GenAI conventions supported by your telemetry library, then add application-owned fields for the prompt release and fingerprint. The supplied OpenTelemetry GenAI semantic-conventions entry point said, when checked on 2026-08-26, that its material had moved and that the page was no longer maintained. Treat the convention version used by your instrumentation as another dependency rather than copying an undated field list into your run schema.

   Do not require unrestricted storage of raw prompts merely to get reproducibility. Where instructions contain sensitive material, retain immutable content in a controlled store and put references plus fingerprints in ordinary telemetry. A fingerprint alone identifies equality but cannot reconstruct content; a retained artifact alone is harder to join to a run. You need both functions somewhere in the system.

   The added cost is telemetry volume, controlled artifact storage, and access management. Logging only the model is the wrong answer: an evaluation rerun against a different assembled prompt is not a reproduction. Logging only the prompt is equally incomplete when the model changed.

4. Hold the other behavior-bearing versions constant

   Prepare a prompt-only candidate by keeping the model, tool schemas, permissions, retrieval configuration, and assembly code fixed. This isolates the change you intend to assess.

   If a tool schema must change with the prompt, release the pair as an agent configuration bundle and record both versions. Do not hide the schema change inside the prompt release. A candidate that calls a renamed tool less often may reflect an unavailable schema rather than better wording.

   Apply the same rule to context handling. If the candidate also changes truncation, summarization, or retrieval order, record that as a separate configuration revision; context-window management can alter which instructions remain available late in a run.

   This discipline costs additional release coordination. It is the wrong choice only when the components cannot operate separately. In that case, test and canary the bundle, but retain separate identities inside it so later diagnosis can distinguish what moved.

5. Evaluate the exact candidate artifact against the current release

   Run evaluations on the immutable assembled artifact, not on the editable source template. Feed the same dataset to the current and candidate prompt while holding the model and other configuration fixed.

   The OpenAI evaluation workflow, checked 2026-08-26, describes three core actions: define the task, run it with test inputs, then analyze the results and iterate on the prompt. Apply that sequence before production routing. Version the task definition, inputs, expected labels or grading rules, and candidate artifact together so the result can be rerun later.

   Cover the decisions the prompt is supposed to change and the behavior it must preserve. For an agent, inspect at least:

   • task result against the evaluation rule;
   • selected tool and tool arguments;
   • actions requiring refusal or human approval under your agent permission model;
   • whether the agent stops after completion;
   • whether it stops prematurely, repeats a tool call, or reaches the configured run limit;
   • output shape where downstream code depends on it.

   Use the same graders for current and candidate releases. If a grader changes, give it a new version and rerun both sides. Otherwise a score difference combines a prompt change with a measuring-system change.

   A fixed dataset cannot cover every production input. Its job is to block known regressions and make release comparisons repeatable, not to certify all future behavior. Production canary measurements cover the traffic distribution the dataset missed. If the dataset itself needs work, separate that task into building representative agent evaluation datasets.

   The concrete costs are model calls, grader execution, dataset upkeep, and review of ambiguous failures. This step is the wrong place for a simultaneous model upgrade: even if the combined result improves, you will not know whether the prompt change caused it.

6. Put prompt selection behind a reversible runtime switch

   Route prompt releases through one decision point that accepts a stable subject, chooses a release ID, and records the decision. The selected ID must resolve to an immutable artifact.

   A prompt rollout needs the same canary and rollback path as code because small wording changes can alter tool selection and stopping behavior. Keep the current production prompt as the default branch and expose the candidate only through the release switch. Operators must be able to set the candidate share to zero without editing the artifact or rebuilding application code.

   Pete Hodgson’s feature-toggle treatment, published 2017-10-09, shows a canary cohort consistently receiving the new path while the rest remain on the old path. Its 1% cohort is an example, not a universal starting value. Choose the initial share from the consequence of failure, available traffic, and time needed to observe your release criteria; the supplied sources establish no generally correct percentage.

   Cohort assignment should remain stable for the unit you evaluate, such as a user, tenant, conversation, or workflow. Reassigning a conversation between prompt versions halfway through mixes two instruction sets in one run history. For long agent runs, bind the chosen release to the run at creation and keep it there.

   Test both switch states and the missing-configuration fallback. State explicitly which prompt wins if the flag service is unavailable. A safe operational default is a project decision, not something to discover during an incident.

   The switch adds routing logic, configuration state, and combinations to test. It is the wrong answer to leave every old prompt path indefinitely. Keep the rollback path through the observation period, then remove stale routes under the same change-control process used for code flags.

7. Canary against prompt-specific failure signals

   Start with the candidate disabled, verify that the current release still resolves, then enable the candidate for the first stable cohort. Compare candidate and current runs over the same interval.

   Decide promotion and rollback conditions before increasing exposure. Include the evaluation outcomes that matter to the task, plus operational counts for tool choice, approval requests, refusals, repeated calls, run-limit exits, and other stopping results your agent records. Do not invent a generic percentage improvement or failure threshold. Set each threshold from the application’s existing tolerance and the consequence of the affected action.

   Inspect traces from both releases, not only aggregate task scores. A candidate can reach the expected final answer by taking a different sequence of actions. That difference is material when tools write data, send messages, spend money, or require approval.

   Increase exposure only after the candidate meets the predeclared conditions. Keep cohort logic, prompt release, model, and graders unchanged during an observation stage. If any of them changes, begin a new stage with a new recorded configuration.

   The cost is slower release progress and enough production traffic to compare cohorts. Canarying is the wrong answer when a single candidate action can cause an unacceptable irreversible result. In that condition, keep the affected tools unavailable or approval-gated until the prompt has been exercised in a non-destructive environment.

8. Rollback by routing, then diagnose from the recorded tuple

   On a rollback condition, set the candidate share to zero and leave the failed artifact unchanged. New runs should resolve to the preceding production release; existing-run behavior follows the run-binding rule established earlier.

   Confirm rollback in telemetry by checking the prompt release ID on new runs. A changed flag value is not sufficient evidence if clients cache prompt artifacts or cohort decisions.

   Diagnose using the complete tuple: prompt release and fingerprint, requested and returned model, assembler revision, policy revisions, tool schemas, rollout cohort, and evaluation version. Re-run the failed input against the retained configuration where your data rules allow it. The same records also support later agent run replay and debugging, although replay should not repeat an external side effect merely to reconstruct the decision.

   Do not edit the failed artifact and retry under the same ID. Create a successor candidate, evaluate it against the unchanged production release, and begin a new canary. That preserves the meaning of the failed run and its evaluation results.

9. Promote the candidate and close the release

   Promotion changes the default routing pointer; it does not rewrite the candidate artifact. Record the promotion time, the release that became production, the release it replaced, the evaluation revision, and the final canary configuration.

   Retain the replaced release until the rollback window defined by your operating process has closed. Then retire the temporary rollout flag and obsolete routing branches. Keep the immutable manifest and run references for as long as your debugging, audit, and data-retention rules require.

   If the model, policy, assembler, or tool schema changes after promotion, create a new configuration identity and run the release process again. A prompt label that silently follows mutable dependencies is not a released version.

Expected result

Done means every model request resolves to an immutable assembled prompt artifact; every run records that prompt identity beside the requested and returned model identities; evaluations name the exact artifacts they tested; production traffic can be split into stable current and candidate cohorts; and operators can route new runs back to the retained prior prompt without a code deployment.