How to Evaluate Non-Deterministic AI in CI

Before you start

You need three things in place, because each step below assumes them.

First, a set of inputs you actually care about — twenty is enough to start, and they should be drawn from real traffic rather than invented at your desk. Second, a way to run those inputs against the system from a CI job without a human in the loop. Third, and least technical: someone with authority to decide what failure rate ships. That last one is a prerequisite, not a formality — step 5 stalls without it, and it is not a decision you should make alone.

Budget matters too. The technique in step 4 multiplies your token spend by the number of repeats, so know what your evaluation run costs before you make it a required check. If you have not instrumented that yet, attributing agent cost and latency to the work that caused it is the prerequisite to knowing whether a 5x eval suite is affordable on every pull request.

Steps

1. Delete every assertion on exact output

Start by removing what you have. A test that compares the model’s response to a stored string fails whenever the wording shifts — a new model version, a reordered list, a synonym. That failure carries no information about whether the system got worse. Martin Fowler’s write-up on non-determinism in tests makes the consequence plain: a suite that cries wolf gets ignored, and once the team is trained to re-run red builds without reading them, the suite has negative value. It is worse than no suite, because it costs CI minutes and buys false confidence.

The tell is a diff where the fix is updating the expected string. If that is the routine repair, the assertion is measuring phrasing, not behaviour.

2. Replace them with property-based assertions

Assert on things that must be true of any correct answer, not on the answer itself:

• the response cites at least one source from the retrieved set
• the JSON parses, and validates against the schema
• no forbidden claim appears — no pricing figure, no medical advice, no competitor named
• required fields are non-empty

Each of these survives a rephrasing and still fails on real breakage. If retrieval silently returns nothing, the citation check goes red. If a prompt change breaks the output contract, the parse check goes red. If a model starts volunteering advice you are legally not allowed to give, the forbidden-claim check goes red. None of them care which words were used to be correct.

The mechanism is the same one behind property-based testing generally: you state an invariant and let the inputs vary underneath it. Hypothesis’s documentation is the clearest treatment of the idea, and although it generates inputs rather than tolerating variable outputs, the discipline of naming the invariant transfers directly. OpenAI’s simple-evals shows the same shape applied to model evaluation — graded checks over sampled outputs rather than string equality.

Write the properties before you look at any output. Writing them afterwards produces assertions that describe what the model happened to do.

3. Set temperature to zero, and do not trust it

Drop temperature to zero for the eval run. It narrows the distribution and removes a large source of noise for free, so there is no reason not to.

But it does not make the system deterministic. Batching, floating-point non-associativity across GPU kernels, load-dependent routing on the provider side, and model version rollouts all reintroduce variation that temperature does not touch. A suite whose design assumes identical output at temperature zero is unsound — it will pass for weeks and then fail on a day when nothing in your repository changed. Treat temperature zero as variance reduction, and build the rest of the suite as though output still varies. If you route across providers or tiers, the assumption is even weaker: two backends will not agree token-for-token, which is the case whether you are routing requests between models or choosing between a frontier and a mid-tier model for a task class.

4. Run each input N times and assert on the pass rate

This is the step that changes the character of the suite. Instead of running an input once and getting a boolean that is partly a coin flip, run it N times and record how many runs satisfied the properties. A single run of a flaky check tells you almost nothing; ten runs tell you the check passes 8 times in 10, which is a number you can track, threshold, and compare across commits. The flaky boolean becomes a measurable one.

The cost is exactly what it looks like: N times the tokens, and N times the wall-clock unless you parallelise. Pick N deliberately. Five gives you resolution to the nearest 20 percentage points, which is coarse but often enough to catch a real regression. Twenty gives you 5-point resolution at four times the spend. There is no free lunch here — the precision of your measurement is bought with tokens.

If you parallelise the repeats, you will hit provider limits quickly at twenty inputs times ten runs; the mechanics of staying inside them are covered in agent execution under provider rate limits and concurrency caps. A practical middle path is a small N on every pull request and a large N on a nightly run, so the expensive measurement happens once a day rather than once a push.

5. Get a tolerance decided, and write it in the config

Now you have a pass rate per input and an aggregate across the suite. The suite needs a number to compare it against — and choosing that number is a product decision, not an engineering one.

“Ninety percent of inputs must satisfy every property” is a statement about how much user-visible failure the business will accept, and it depends entirely on what the failure does. A summarisation feature that occasionally omits a detail and a system that occasionally emits a forbidden claim do not get the same threshold, and no amount of engineering judgement derives one from the other. Take the numbers to whoever owns the product, show them what a failure at 90% actually looks like in output, and have them pick.

Then write it down where the suite reads it — a threshold in a config file, not a number buried in an assertion. When the product owner changes their mind, the diff should be one line and legible to them.

One refinement worth making early: not every property deserves the same tolerance. Forbidden-claim checks are usually a hard zero, and format checks near-100%, while quality-shaped properties get the negotiated number. Splitting the thresholds by property class is cheaper than arguing about a single global figure. If your properties are graded by another model rather than by code, the tolerance question compounds — see golden sets versus model-as-judge for how that changes what you are measuring.

6. Wire it into CI as a gate, not a report

Run the suite on every change to prompts, model version, retrieval configuration, or tool definitions — the inputs that actually move behaviour. Fail the build when the aggregate pass rate falls under the tolerance from step 5.

Store the per-input pass rates from each run. The absolute number matters less than the trend: an input that has run at 100% for a month and drops to 70% is a signal even if 70% is above your gate. Without stored history you can only see the current snapshot, and regressions that stay inside tolerance will pass silently until they compound.

Expected result

You have a CI check that goes red when the system’s behaviour degrades and stays green when the wording changes. Failures name the property that broke and the pass rate that fell short, so the person reading the build knows what to look at without re-running anything.

The suite’s tolerance lives in a config file, was chosen by whoever owns the product, and can be changed in one line. Each input’s pass rate is stored per run, so a drift from 100% to 80% is visible before it crosses the gate.

The cost is fixed and known: N times your per-input token spend, on a schedule you chose. And the failure mode you started with — a red build the team re-runs without reading — is gone, because the assertions no longer fire on rephrasing.