Build Representative Agent Evaluation Datasets

Prerequisites

Before collecting cases, you need access to agent run records, a reproducible test environment, and reviewers who can decide whether an outcome is acceptable. Logs that contain only the final answer are insufficient: preserve the request, starting state, exposed tools, tool results, and outcome whenever your privacy rules allow it.

Write down the decision the evaluation will support. Examples include releasing a prompt revision, changing a tool schema, replacing a model, or accepting an agent-state migration. The decision matters because it determines which cases, labels, and failure costs belong in the dataset.

The OpenAI guide to working with evals, accessed August 26, 2026, describes an evaluation as test inputs plus criteria for judging the output. It also says the test data should represent the data the application is expected to handle. For an agent, extend that input beyond the user message: the environment and available actions are part of the test.

Steps

1. Define the unit of evaluation

   Treat one case as one complete decision context, not one prompt. An agent evaluation case needs the initial state and available tools as well as the user request because those inputs determine the action path.

   Define the end of a case before collecting examples. It might end when the requested artifact exists, the agent asks for approval, a required tool fails, or a maximum-step policy stops the run. The definition must match the production boundary you care about. If production evaluates a completed workflow but the dataset grades only the first response, the test cannot reveal failures introduced by later tool calls.

   Record what the evaluation is intended to detect: an incorrect final result, an unsafe action, a missing approval, an unnecessary tool call, failure to stop, or failure to recover from an allowed tool error. Do not merge these into a vague quality label. They have different mechanisms and may require different graders.

   Use the NIST AI Risk Management Framework, accessed August 26, 2026, as a scope check rather than a scorecard. NIST describes the framework as voluntary guidance for incorporating trustworthiness considerations into the design, development, use, and evaluation of systems. Translate the risks relevant to your product into observable case fields and pass conditions. A risk that has no case, label, or review rule is not being tested.

   This step costs engineering and reviewer time because the team must agree on boundaries before generating results. It is the wrong place to preserve every detail merely because it appeared in a trace. Keep only context that can affect the path, the judgment, or the ability to reproduce the case.

2. Create a case schema that can reconstruct the choice

   Define the schema before sampling. Otherwise, early cases accumulate whatever happened to be easy to export, and later cases cannot be compared cleanly.

   A practical case record needs these fields:

   Field	What it records	Why it belongs
   case_id	Stable identifier	Keeps reviews, failures, and revisions attached to the same case
   provenance	Production, authored, imported, or transformed	Separates observed demand from deliberate coverage
   user_request	Exact or privacy-reviewed request	Establishes the requested outcome
   initial_state	Relevant files, records, messages, permissions, and prior state	Reconstructs what the agent knew and could affect
   available_tools	Tool names, input schemas, scopes, and relevant failure fixtures	Reconstructs the possible action path
   expected_outcome	Required end state or answer properties	Defines success without forcing one exact trajectory
   allowed_actions	Actions that may be taken	Prevents a correct result reached through an unacceptable operation from passing
   forbidden_actions	Actions that must not be taken	Makes safety and approval boundaries testable
   grader	Deterministic check or reviewer rubric	States how the result becomes a verdict
   risk_tags	Failure modes represented by the case	Supports coverage review and segmented results
   privacy_status	Review state and permitted use	Blocks unreviewed production material from entering the suite
   split	Development or holdout	Prevents accidental tuning against reserved cases
   fixture_version	Version of state and tool contracts	Makes later replay failures diagnosable

   Store state as the smallest fixture that still preserves the decision. A filesystem agent may need file contents, paths, permissions, and repository status; it does not automatically need an entire production checkout. A support agent may need the relevant conversation and account state; it does not automatically need every historical message.

   If persisted state changes shape, apply the same compatibility discipline used for agent-state schema migrations. A case that silently loads under a different schema no longer represents the original starting condition.

   Full trace capture is useful during diagnosis but expensive to retain and hard to review. Minimal fixtures are cheaper and clearer but become the wrong answer when removing a field changes tool eligibility, authorization, or the expected outcome. Test minimization by replaying the case after each reduction.

3. Inventory the production task distribution before selecting cases

   Build a candidate inventory from actual task categories, environments, tool combinations, outcome types, and known failure modes. The purpose is not to create the largest list. It is to expose what the agent is actually asked to do and the conditions under which those requests arrive.

   Separate three sources of cases:

   • Production cases show what users and systems actually generate. They track demand, but they inherit logging gaps, privacy constraints, and whatever the current agent already fails to reach.
   • Authored cases target boundaries that may be rare but important, such as an unavailable tool, missing approval, ambiguous state, or conflicting instructions. They give controlled coverage, but their wording and assumptions can reflect the author more than production.
   • Imported benchmarks provide an outside reference point. They are useful only when their tasks, tools, and scoring boundary match your agent closely enough to inform the release decision.

   The GAIA benchmark paper, submitted November 21, 2023, illustrates why agent tasks cannot be reduced to isolated answers. Its real-world questions require combinations of reasoning, multimodal handling, web browsing, and tool use. The paper contains 466 questions and withholds answers for 300 of them for its leaderboard. Its reported 92% human result and 15% result for GPT-4 with plugins belong to that benchmark and setup; do not transfer either number to your agent.

   Importing GAIA or another benchmark is the wrong answer when your agent operates with different tools, state, permissions, or success criteria. In that condition, use the benchmark as inspiration for task structure, not as a substitute for product cases.

   Do not claim the inventory is representative merely because it is varied. Compare it with the production categories you can observe and mark gaps as unverified. If telemetry cannot establish frequency, retain the category but do not invent a weight.

4. Turn candidate runs into reproducible fixtures

   For each selected run, reconstruct the initial state before the first agent action. Include the tool surface exactly as the agent saw it: tool names, descriptions, input contracts, authorization scope, and any relevant availability condition. A request tested with a broader tool set is a different case because it permits a different path.

   Replace live dependencies with controlled fixtures when the release question does not require the live system. A fixed search result, file tree, or API response makes the case repeatable and keeps unrelated external change from altering the verdict. Preserve error responses when recovery is part of the behavior under test.

   Do not over-specify the expected action sequence. If several safe paths can reach the same correct state, grading one recorded trajectory will reject valid behavior. Specify required and forbidden actions, required evidence, and the final state instead. Require an exact trajectory only when order itself carries the risk, such as obtaining approval before a side effect.

   Link fixtures to the machinery for agent-run replay and debugging. Replay is the check that the stored request, state, tools, and responses are sufficient. If a case cannot be replayed without fetching undeclared context, its fixture is incomplete.

   Fixture construction costs storage, maintenance, and adapter code. Live tests cost less fixture work but introduce external variation and may create real side effects. Use live dependencies only when that variation is the subject of the evaluation and the actions are safely contained.

5. Admit production failures only after privacy review

   Production failures should enter the dataset after privacy review so the suite tracks the distribution that actually matters. This is an admission pipeline, not a manual copy-and-paste habit.

   First, identify the smallest replayable slice of the failed run. Then send the request, state, tool inputs, tool outputs, and any human feedback through the privacy process required for that data. Record the review result in privacy_status. Only approved material moves into the evaluation store.

   Apply approved transformations consistently. If policy requires redaction, replacement, or minimization, preserve the relationships that caused the failure. Replacing an identifier is acceptable when identity is irrelevant; removing the distinction between two permissions is not acceptable when that distinction caused the wrong tool call. After transformation, replay the case and confirm that the same failure remains possible.

   Attach the observed failure category and the corrected expected outcome. Human feedback is valuable when it includes the context behind the judgment; a bare thumbs-down does not say whether the result, action path, tone, or approval behavior was wrong. Keep that context using the same principles as capturing human feedback on agent runs.

   Raw production logs are the wrong answer when they have not passed review, when the required consent or authority is absent, or when de-identification destroys the mechanism being tested. In those cases, author a synthetic case that expresses the mechanism and label its provenance honestly. Do not describe that synthetic case as observed production evidence.

6. Add deliberate coverage without falsifying frequency

   Production sampling misses cases that are important but uncommon, newly possible, or suppressed by the current interface. Add authored cases for those boundaries, but keep them distinguishable from production-derived cases.

   Build each authored case around one change from a known baseline: remove a required tool, deny a permission, make state stale, return a tool error, introduce an ambiguous request, or require approval before mutation. The single-change structure identifies which condition produced the different behavior. A case that simultaneously changes the request, state, tools, and grader may fail without explaining why.

   Include controls. If a case tests refusal when a tool is unavailable, retain a paired case in which the tool is available and the request should succeed. If a case tests a prohibited action, include a permitted neighboring action. These pairs distinguish a narrow policy from blanket refusal.

   Authored edge cases cost maintenance because they depend on product rules and tool contracts. They are the wrong basis for estimating production pass rates unless their weights come from observed production data. Report their results as targeted coverage instead.

7. Label outcomes and action constraints separately

   Give each case a pass rule that can be applied without reading the author’s mind. Split the judgment into final outcome, action-path constraints, and stopping behavior.

   Use deterministic checks when the requirement is machine-observable: a file has the expected content, a record remains unchanged, a required field exists, or a forbidden tool was never called. Use a rubric when correctness requires judgment, but write the rubric as observable conditions. Record whether the reviewer needs the request, state, trace, final answer, or all four.

   Ground truth does not always mean one exact response. For a state-changing agent, the required final state may be the ground truth while several messages and tool sequences are acceptable. Conversely, a correct final state is not enough when the agent exposed private data, skipped approval, or performed an unnecessary destructive action on the way there.

   Review ambiguous cases before placing them in the holdout. If qualified reviewers can reasonably disagree because the requirement is missing, repair the case instead of averaging away the ambiguity. A difficult case is useful; an underspecified case is noise.

   Detailed manual rubrics cost reviewer time. Exact-match grading is cheaper but is the wrong answer for tasks with multiple valid outputs or paths. Prefer the cheapest grader that still measures the actual requirement.

8. Remove leakage, duplicates, and accidental shortcuts

   Compare cases by more than exact text. The same production incident may appear with a shortened request, a different identifier, or a lightly edited fixture. If one version lands in development and another in holdout, the holdout no longer tests an unseen situation.

   Look for shortcuts in labels and fixtures. A filename, case identifier, redaction marker, or tool response should not reveal the expected verdict unless that signal exists in production. Remove author notes and answer-bearing metadata from the agent-visible input.

   Keep legitimate repeated demand when frequency is known and the evaluation is intended to reflect it. Deduplication is the wrong answer when it erases the fact that one task dominates production. Instead, retain one canonical fixture plus an explicit, evidence-backed weight. If frequency is unknown, do not manufacture weighting.

   Also check temporal leakage. A case based on a production failure must not expose the later correction as part of its initial state. The fixture should begin where the original decision began.

9. Reserve the holdout before prompt tuning

   Split cases only after privacy review, fixture validation, labeling, and deduplication. A holdout set reduces the chance that prompt tuning optimizes only the examples the author repeatedly inspects.

   Keep the holdout inaccessible during ordinary prompt iteration. Developers tune against the development set; a separate release path runs the holdout and reveals aggregate and failure results at the decision point. If authors repeatedly inspect a holdout case and modify the prompt in response, move that case into development and replace it with a genuinely unseen case.

   Split related families together. Variants of the same incident, paired controls, and transformations of one source case should not cross the boundary. Otherwise, the development member teaches the condition tested by its holdout sibling.

   No supplied source establishes a universal split percentage, so choose the holdout size from the number of independent case families, the cost of review, and the release risks you need to detect. Do not present an arbitrary percentage as evidence-based.

   A fully hidden holdout costs diagnostic speed because failures are not available during every edit. It is the wrong answer as the only suite: teams also need visible development cases for rapid debugging. Use both sets, and connect prompt changes to versioned agent prompt releases so every result names the prompt it evaluated.

10. Version the dataset and define its admission loop

Version the schema, fixtures, labels, graders, and split assignments together. A pass-rate change is uninterpretable if the agent and dataset changed without separate records.

For every dataset revision, record added and removed case IDs, provenance, privacy status, fixture versions, label changes, and the reason for each change. Do not silently rewrite a failing case to match current behavior. If the old expectation was wrong, preserve that correction in the history.

Establish an operating loop: review production failures, complete privacy review, build the minimal fixture, label it, check for related cases, assign its case family to the appropriate split, and include it in the next dataset version. This is how the suite continues to track the failures users actually encounter instead of freezing around launch-day assumptions.

Run the complete dataset against a pinned agent configuration before using it for a release decision. Record the model, prompt version, tool contracts, state schema, fixture version, graders, and execution policy. Route the resulting measurements into explicit agent regression release thresholds rather than deciding after seeing whether the aggregate score looks comfortable.

Constantly rewriting the suite is the wrong answer when it prevents comparison across releases. Preserve stable cases while adding new production evidence, and make a breaking dataset revision explicit when the product boundary changes.

Expected result

Done means every evaluation case contains a user request, reconstructable initial state, declared tool surface, expected outcome, action constraints, grader, provenance, privacy status, fixture version, and split assignment. Production-derived failures have documented privacy approval and still reproduce after transformation. Related cases do not cross into the holdout, prompt authors do not tune against holdout contents, and each evaluation run identifies the exact agent and dataset versions used.