Keep an Agent's Context Useful on Long Runs

Prerequisites

Before changing the loop, make one complete run inspectable. You need to see the instructions, tool definitions, retrieved material, messages, tool calls, and tool results assembled for each model turn. Token totals alone are not enough: they show how much context was sent, not whether it helped.

You also need an editable state file that belongs to the run, plus a small evaluation set containing instructions whose compliance can be checked. Use tasks representative of production, including at least one with several tool calls and one in which an early conclusion is corrected later.

Anthropic’s agent-building guidance, published 19 December 2024, describes agents as models using tools in a loop, checking environmental feedback as they proceed. That loop is the unit to fix. A larger framework may make context assembly harder to inspect; if so, expose the final request at the provider boundary before tuning anything else.

Steps

1. Account for the three consumers of context.

   Treat context as a budget spent on instructions, retrieved material, and history. Only the first is under the prompt author’s direct control. The retrieval layer decides which documents and tool results enter; the loop decides which previous messages remain. Editing the system prompt while ignoring those two paths fixes only one part of the request.

   Record the size of each category separately on every turn. Count the full instructions, including policy fragments added by middleware; the retrieved passages and tool results; and all retained user, assistant, and tool messages. Keep tool definitions as a separate line item even if your provider reports them within input tokens. Their special loading behaviour matters in the next step.

   Size is an accounting measure, not a quality score. A long instruction can be necessary, and a short retrieved passage can be irrelevant. The purpose of this inventory is to show where growth originates and which component owns the fix.

   The cost is instrumentation work and the storage required for traces. If requests contain secrets or personal data, retain counts and redacted structure rather than copying raw content into a broadly accessible telemetry system. This accounting is the wrong stopping point when instruction following is already slipping: continue through the relevance and evaluation steps instead of declaring the run healthy because it remains below the model’s advertised capacity.

2. Remove tools and MCP servers that the run will not use.

   Tool definitions are loaded before the first message, so an unused server’s schemas are paid for on every turn of every conversation. The run incurs that input burden even if the agent never calls the server. Ten speculative integrations are therefore not dormant configuration; they are permanent context residents.

   Build tool sets by task or phase. A research phase might expose search and document-reading tools. An implementation phase might expose repository editing and tests. A release phase might expose deployment checks. Select the set in code before the model call. Do not ask the model to ignore twenty irrelevant tools: their schemas have already entered context by the time it reads that instruction.

   Within each selected tool, keep names, descriptions, parameters, boundaries, and error behaviour explicit. Saving context by making a schema ambiguous shifts the cost into failed calls and corrective history. Prefer one clear operation over several nearly identical operations whose selection rules require a paragraph of explanation.

   The engineering cost is a registry or routing layer and tests proving that each phase receives the tools it needs. Dynamic selection is the wrong answer when a safety-critical tool must remain visible at every decision point, or when routing could hide the only recovery operation. In those cases, keep the required tool loaded and remove optional ones around it. A plain tool-calling loop versus an agent framework is partly a context decision: choose the abstraction whose assembled request you can inspect and control.

3. Put an admission gate in front of retrieved material and tool results.

   Relevance degrades faster than capacity fills. A window holding fifty tool results can be technically fine and practically a worse reasoner than one holding five. The problem is not overflow. It is competition among current evidence, obsolete evidence, duplicate evidence, errors, and instructions.

   Give every result a reason to enter the next turn. Admit it if it answers the current subproblem, changes a decision, supplies evidence the agent must quote or transform, or reports the state of an action just taken. Otherwise, keep it in the trace or an external artifact and omit it from the model request.

   Apply the gate before concatenation. Reject malformed, truncated, contradictory, or unexpectedly large results in code. Extract the fields the next step needs instead of sending a complete response merely because the tool returned it. Preserve a reference to the raw artifact so the agent can request it again if the omitted detail becomes relevant. This complements validating tool results before they enter model context: structural validation prevents bad data from entering, while admission control prevents valid but irrelevant data from staying.

   Do not collapse different results into a single undifferentiated paragraph. Keep the source, retrieval time, query or tool call, and result status attached. When two results conflict, carry the conflict explicitly rather than selecting one silently.

   The cost is application logic, possible extra retrieval calls, and latency when the agent must reopen an artifact. Admission control is the wrong answer when the task genuinely requires comparing the complete set at once. In that case, partition the comparison into focused calls and combine their checked conclusions, rather than pretending all fifty results are equally salient in one turn.

4. Move conclusions into an editable run-state file.

   Writing conclusions to a file and re-reading the file is more reliable than carrying them in history, because a file can be edited when a conclusion turns out wrong. History is append-only evidence of what the agent once believed. If a later tool result disproves an early assumption, both versions remain in the transcript and compete for attention.

   Create one canonical state file per run. Keep it concise but structured around the decisions later steps actually consume:

   • the current objective and completion conditions;
   • confirmed constraints;
   • decisions and the evidence supporting each one;
   • open questions and blockers;
   • artifacts created or changed;
   • rejected options and the reason for rejection;
   • the next action.

   Require the loop to read this file at the start of each phase and update it after a decision changes. An update should replace a disproved conclusion, not append another narrative paragraph beneath it. Keep the trace as the audit record; use the state file as the current operational truth. The distinction is central to deciding what an agent should persist between steps and runs.

   For example, if the agent first records deployment target: preview and later receives approval for production, edit that field and preserve the approval reference. Do not leave both targets in a chronological summary and expect the model to infer which one governs the next tool call.

   The cost is file I/O, schema design, write-conflict handling, and validation before a state update becomes authoritative. A shared mutable file is the wrong answer for parallel workers writing unrelated branches. Give each worker isolated state, then have one coordinator merge checked conclusions into the canonical file. It is also the wrong place for raw tool output; store large artifacts separately and reference them.

5. Compact history only at explicit checkpoints.

   Summarising history compresses it lossily, and the loss is systematically biased toward dropping the specifics that later steps need. A summary tends to preserve the apparent storyline while losing an exact filename, rejected parameter, boundary condition, evidence reference, or reason a tempting option failed. Those details look minor while summarising and become decisive later.

   Compact after a completed phase, not merely because a turn counter fired. First update and validate the state file. Then replace the completed phase’s conversational history with a checkpoint record containing its objective, actions, conclusions, unresolved issues, changed artifacts, and exact references needed to recover evidence. Keep recent messages that define the active subproblem verbatim.

   Do not ask one free-form summarisation prompt to decide what matters. Supply fields and reject a checkpoint that omits required ones. If a detail can be regenerated cheaply and deterministically, a reference may be enough. If it records human approval, a safety constraint, or a failed attempt that must not be repeated, preserve the exact operative detail.

   The cost is a summarisation call or deterministic extraction pass, plus the chance of introducing a false statement during compression. Validate checkpoint fields against the state file and trace. Compaction is the wrong answer in the middle of a tightly coupled debugging exchange where the exact sequence of observations matters. Finish or pause that subproblem first. It is also wrong as a substitute for retrieval filtering: repeatedly summarising irrelevant results still leaves irrelevant material in charge of the summary.

6. Start a fresh context at phase boundaries.

   Once a phase has a checked state file and checkpoint, assemble the next turn from the stable instructions, the current state, the active objective, and only the evidence required for that objective. Do not keep the entire transcript out of habit.

   This reset works because the next phase reasons from current, editable conclusions rather than reconstructing them from chronological conversation. It also makes context growth bounded by the active phase instead of the whole run. The full trace remains available for audit and targeted re-reading.

   Define phase boundaries from the work: research complete, plan approved, implementation complete, verification started, or release decision requested. Avoid a universal turn or token threshold. The supplied evidence contains no measured threshold at which every task degrades, and the fifty-versus-five example shows why capacity alone is the wrong trigger.

   The cost is orchestration code and occasional rereads when the checkpoint omitted something. A reset is the wrong answer before the state has been validated, because it can turn a compression mistake into the only context the model sees. It is also wrong when the next action depends on exact conversational wording that was never written to durable state; preserve that wording or delay the reset.

7. Apply prompt caching after context hygiene.

   Put genuinely stable prefix material first: core instructions, the selected tool definitions, and any invariant task policy. Keep changing state, retrieved evidence, and the current request after that prefix. This layout may improve cache reuse where the provider supports it.

   Prompt caching changes the economics rather than the limits. A cached prefix is cheaper to re-send, not more useful to the model. It still occupies the request and still competes for attention. Caching an unused server schema or stale policy makes repeated waste less expensive; it does not make that material relevant.

   Measure cached and uncached input separately, but keep total assembled context and instruction-following results beside those cost measures. Treat prompt caching for repeated agent turns as a billing and latency optimisation after the tool and instruction set is correct.

   The cost is prefix stability work, provider-specific integration, and cache misses whenever early material changes. Caching is the wrong answer for rapidly changing prefixes or low-reuse runs. It is also the wrong remedy for context drift: remove or relocate irrelevant material even when resending it is cheap.

8. Measure instruction following as the leading failure signal.

   The measurable symptom of a full window is not an error but a drop in instruction-following, which is why it gets diagnosed as a model problem. The request can remain valid and the provider can keep returning successful responses while the agent skips a required check, revisits a rejected option, uses the wrong tool, or treats an obsolete conclusion as current.

   Instrument each model call with the run and turn identifiers, phase, selected tool set, input and output usage, cache usage when available, tool calls, and evaluation results. Start with OpenTelemetry’s Generative AI semantic-conventions index, checked 26 August 2026. That URL now marks the GenAI material as moved, so pin the convention revision your instrumentation implements instead of assuming the index remains a stable schema.

   Keep message content out of default telemetry when counts and references are sufficient. If you capture content for debugging, apply your existing access and retention controls; the context-maintenance design does not require every prompt to become a permanent log.

   Build paired evaluations from the same task: one at a clean phase start and one after controlled history or tool-result accumulation. Score observable requirements separately, such as whether the agent used the required source, obeyed a forbidden-action rule, validated before writing, respected the current state file, and stopped at the defined condition. Do not hide these checks inside one subjective quality score.

   The Martin Fowler collection of dated field reports on generative-model development, checked 26 August 2026, includes accounts of assistants getting stuck in prior work, the importance of developer feedback loops, and unreliable output across tasks. Use such reports to choose failure modes worth testing, not as benchmark data for your system. This page supplies no productivity or context-capacity benchmark.

   Set release thresholds for each critical instruction and follow the same policy used for agent evaluation regression thresholds. When performance falls with accumulated context but recovers after a clean restart using the same model and state, investigate context assembly before changing models.

   The cost is a representative evaluation corpus, repeated calls, trace analysis, and maintenance when prompts or tools change. Instruction-following evaluation is the wrong answer if the assertions cannot be observed. Rewrite vague goals such as be careful into checkable actions, or keep them out of automated scoring.

9. Turn the policy into run-time controls.

   Enforce the design in the loop rather than relying on an instruction that asks the agent to manage its own context. Before every call, select tools, admit evidence, load current state, assemble the active history, and record category sizes. At a phase boundary, validate state, write a checkpoint, and start a fresh context. Apply caching only after that assembly is correct.

   When an evaluation detects instruction drift, stop adding history. Preserve the trace, compare the active request with a clean phase-start request, and identify whether instructions, retrieval, tool schemas, or history introduced the competition. Correct the state file if it is wrong; remove irrelevant material if it is merely old; reopen exact evidence if compaction lost a necessary detail. Resume from a checked phase boundary.

   The cost is stricter orchestration and more explicit failure handling. These controls are the wrong answer for a single-call workflow with no accumulated history. For a genuinely long agent run, leaving context management to the same crowded context creates the failure you are trying to prevent.

Expected result

Done means each turn contains the smallest complete context for its current decision: stable instructions, only the tools available in that phase, admitted evidence, an editable current-state record, and limited active history. Completed phases remain recoverable through checkpoints and traces without occupying every later request.

A long run no longer waits for a context-window error. It detects degradation through instruction-following evaluations, can attribute growth to instructions, retrieval, tools, or history, and can restart from checked state when relevance falls. Prompt caching may reduce the cost of the stable prefix, but it does not alter that quality standard.