Prompt Caching for Repeated Agent Turns

Prompt caching strategy

Prompt caching reuses model-processing work when successive agent requests begin with the same token sequence. The practical strategy is to keep a long, stable prefix at the front of every turn, append changing material after it, and measure how many input tokens are actually reused.

Reuse follows the prompt prefix

A prompt cache is not a lookup for similar requests. It preserves processing state for an exact prefix. The OpenAI prompt-caching guide, checked 2026-08-26, describes the cached material as key-value state produced while the model processes input tokens. A later request can reuse that state when its rendered prefix matches an eligible cached prefix; the model still processes everything added after the match.

That left-to-right mechanism determines the prompt layout. Put the material shared by many turns first and the material specific to one turn last. For a tool-using agent, the rendered request commonly has this shape:

stable tool definitions
stable standing instructions
stable reference context
previous conversation and tool results
current user message or event

The boundary is about the rendered token sequence, not the application objects that produced it. Two requests can contain logically equivalent instructions yet fail to share the same full prefix if an earlier part is rendered differently. Conversely, appending a new message preserves the prefix formed by the earlier turns.

Cache reuse does not mean the model skips the prompt semantically. The saved processing state represents the preceding tokens, and the new suffix is evaluated in that context. Caching changes the work needed to process repeated input; it does not create agent memory, alter the context window, or make omitted information available.

Stable tools and instructions come first

Prompt caches reward stable prefixes, so tool definitions and standing instructions should precede volatile conversation content. Anthropic’s prompt-caching documentation, checked 2026-08-26, explicitly places static tool definitions, system instructions, context, and examples at the beginning. It describes its prefix hierarchy as tools, then system content, then messages.

This order gives the most frequently reused material the widest reach. A shared tool schema at the beginning can be reused across many turns and conversations that use the same agent configuration. Conversation messages, retrieved records, tool results, timestamps, and request-specific instructions belong after that shared material because they change more often.

Stability includes ordering and serialization, not only meaning. Renaming a tool, changing its description or schema, reordering tools, or modifying an early standing instruction changes the rendered prefix. Tool registries therefore need deterministic ordering, and prompt assembly should not inject request-specific values into the stable layer merely because the application labels that layer a system prompt.

The design has a concrete maintenance cost: a release that changes early tools or standing instructions starts a new reusable prefix. That miss is expected when the change is necessary. Avoiding all prompt changes would preserve reuse at the expense of correctness, which is the wrong trade. Version the configuration, expect the new version to warm separately, and judge the change using observed tokens and cost rather than hit rate alone.

A long conversation can still extend the reusable prefix when each turn appends messages without rewriting earlier content. Summarization or compaction may reduce total input while replacing an earlier portion of that sequence. The next request can then reuse less of the old prefix even though the shorter request may cost less overall. That is why cache performance and context-window management need to be evaluated together.

One early change cuts off later reuse

Changing one early token can invalidate reuse for everything after it even when most of the prompt is identical. The unchanged material before that token may still match an earlier eligible prefix, but the remainder cannot reuse state derived from a different preceding sequence.

The position of a change therefore matters as much as its size. A one-token edit near the front can discard reuse across tool definitions, instructions, conversation history, and the current turn that follow it. A much larger new tool result appended at the end leaves the established prefix intact.

This is also why a stable cache key or conversation identifier is not proof of a cache hit. Such identifiers may group requests for routing, but they cannot make different rendered prefixes identical. Prefix identity remains the deciding condition.

Inspect the final request sent to the model when reuse falls after a deployment. Compare the rendered prefix, including tool order, tool schemas, standing instructions, output-format instructions, and any context rewritten by compaction. Comparing only source templates can miss a dynamic value or serializer change introduced during request construction.

Treat intentional early changes as configuration boundaries in telemetry. Recording the prompt or tool-set version beside token usage separates expected cold traffic after a release from unexplained variation within one version. If a prompt-layout change can also alter agent behavior, it belongs under the same release thresholds used for agent regressions, not under cost review alone.

Measure cached tokens beside total input

Cache hit rate should be measured beside total input tokens because a high percentage on a small prefix can save less than expected. The useful token-weighted rate is:

cache hit rate = cached input tokens / total input tokens

Retain both terms rather than storing only the result. A percentage loses scale: the same hit rate can represent a small reused prefix on short requests or a much larger number of reused tokens on long requests. The absolute cached-token count shows how much input was eligible for the provider’s cached-input treatment; total input tokens show the workload against which that reuse matters.

Aggregate the numerator and denominator before dividing:

overall hit rate = sum(cached input tokens) / sum(total input tokens)

A simple average of per-request percentages gives a short request the same weight as a long one and can misrepresent the workload. Keep request-level values for debugging, but use summed tokens for route, model, prompt-version, tenant, and time-window reporting.

Provider usage fields need normalization before aggregation. OpenAI reports total input tokens and cached tokens within input-token details. Anthropic reports cache-read tokens, cache-creation tokens, and uncached input tokens separately; its documented total is the sum of those three fields. Map those responses into explicit internal fields rather than assuming that every provider’s property named input_tokens has the same denominator.

For cross-provider traces, start from the OpenTelemetry GenAI semantic-conventions entry point, checked 2026-08-26. That page currently says the GenAI conventions have moved to a separate repository and are no longer maintained at the supplied location. Treat the convention and attribute names as versioned, and retain the raw provider usage response when a common field cannot express a provider-specific cache write or read category.

A useful record for each model call contains:

• total input tokens;
• cached input tokens read;
• cache-write tokens, when the provider reports them;
• uncached input tokens;
• model and provider;
• prompt or tool-set version;
• observed input cost and latency.

Cache-read tokens alone do not establish savings. Cache writes may be billed differently, provider eligibility and retention rules can change, and the uncached suffix still requires processing. Use the token categories with the dated price and usage rules that applied to the request. The broader accounting belongs in cost and latency attribution, where cache behavior can be tied to the agent turn and work that produced it.

What to check next

After the prefix layout is stable, check how conversation compaction changes the rendered request, whether each provider’s usage fields are being normalized correctly, and whether prompt releases produce expected cold-cache periods. Then connect those measurements to per-turn cost and latency, and keep behavioral evaluation beside caching changes so a cheaper prompt is not accepted when it makes the agent less reliable.