How do I know if my evaluation results are contaminated?

Check if the success criteria were documented before the evaluation script was executed. If you decided what counted as a pass after looking at the transcripts, your results are contaminated by your own judgment.

Why should I exclude infrastructure failures from my denominator?

Infrastructure runs, where the agent never produced a line of audio, distort your behavioral metrics. However, you must log these exclusions loudly per test arm. Silently dropping runs is a common way to accidentally hide regressions.

What is an acceptance table in prompt evaluation?

It is a fixed set of contract clauses written before a test matrix runs. Each clause strictly resolves to met, partial, or revert based on predefined percentage bands, eliminating subjective grading.

Why must the scorer run its own self-test first?

The script that scores transcripts is brittle and often fails on predictable phrasing changes. Validating the instrument against known baselines ensures that a failed agent run is actually an agent bug, not a scoring script error.

Write the Acceptance Table Before You Know the Numbers

Symptoms

A prompt or model change ships to production because the person who ran the experiment decided, after seeing the output, what counted as success.
The pull request contains a convincing write-up justifying why certain failures were excluded from the final evaluation metrics.
Six weeks later, a behavioral regression shows up in production that the test suite technically ran over but did not catch.
You find yourself looking at batch evaluation results and deciding a specific row clearly does not matter, or a denominator obviously should exclude a certain set of runs.

What is actually happening

Reducing a sweep to a decision by hand, at the exact moment the data lands, contaminates every threshold with knowledge of the data. It does not feel like cheating. It feels like judgment. You look at a failing transcript and decide the user input was nonsensical, so the agent’s failure to handle it gracefully is acceptable. You look at a small improvement in task completion and decide it is clearly larger than the noise floor.

I did exactly this once and wrote the postmortem on it. When you analyze a matrix of test results without a prior plan, your judgment protects the hypothesis you set out to prove. You dismiss unexpected regressions as irrelevant noise. The fix is not to ask for more discipline in the moment, because the moment is the problem. You cannot unsee the results, and you cannot objectively set a threshold after you already know what number you need to hit to get your branch merged.

How to confirm it

Open the last merged pull request that altered your voice agent prompt and look for a pre-registered acceptance table. Expect to find only the final results, which confirms the pass criteria were decided post-hoc.
Review the evaluation script for hardcoded run IDs or transcript substrings skipped during the aggregation phase. Expect to find silent exclusions added after the initial run to make the script pass.
Compare the total number of runs requested in your batch suite against the denominator used to calculate the final percentage. Expect a mismatch where failed runs were dropped without an explicit rule, confirming denominator manipulation.

The fix

Make the data reduction an artifact that exists before the run. A reducer turns a sweep into a fixed table of contract clauses. Each clause resolves to met, partial, or revert against bands written while the outcome was still unknown. Running it is entirely mechanical. No judgment is exercised after the data exists, because there is nothing left to decide.

If you are running a test matrix across 123 live interviews with 3 repeats each, you define exactly what percentage of successful resolutions constitutes a pass before you trigger the developer-invoked suite. You register the specific measurements that will kill the change if they move the wrong way, regardless of how good the headline number looks. Three of my own prompt fixes were reverted by falsifiers I registered before running them (for more on this, see /blog/voice-agent-prompt-change-improved-one-behavior-but-broke-another/).

There are two supporting rules required to make this work. First, validate the instrument before the subject. The scorer’s self-test must run before any sweep the table will read. Second, exclude infrastructure runs explicitly. An infrastructure run is one that never produced a single agent line. These must be excluded from every behavioral denominator, accompanied by a loud warning if any survive the filter. A silent exclusion is just another place to hide a decision.

What I would check first

Check the git timestamps of the test thresholds against the evaluation output logs. If the thresholds were committed after the logs were generated, it rules out the integrity of the evaluation entirely, as the test was contaminated by the results.
Check the denominator in the final report against the total invoked runs. If the reported denominator is smaller than the total invoked runs, check whether unlogged infrastructure drops silently filtered out failing turns.
Check the exit status of the scorer’s pre-sweep self-test. If the scorer fails on known baselines, it rules out the agent being the source of the regression.

Borrowing from clinical trial practice

This approach generalizes to any change whose outcome is a distribution rather than a boolean. Prompt edits, model upgrades, turn-detection tuning, and retrieval changes all produce varied outcomes across a large test surface.

I borrowed this wholesale from clinical trial practice, and it works for the exact same reason. In a clinical trial, the statistical analysis plan is worthless if it is written by someone who has already seen the patient results. The bands must be locked in advance. In voice agent engineering, the acceptance table acts as your analysis plan. If you edit prompt instructions, you must define the bands for every contract clause before executing the sweep.

Structuring the contract clauses

A contract clause is a specific, measurable condition that the new distribution must meet. A contract clause specifies an explicit, measurable condition evaluated mechanically against predefined bands. For example, a clause sets explicit percentage thresholds for task success that resolve mechanically to met, partial, or revert.

Every clause resolves strictly to met, partial, or revert. A revert outcome on any primary clause halts promotion of the change and forces a revert. There is no negotiating with the reducer script once it finishes. If the results land in the revert band, the branch is discarded or reworked.

Validating the instrument independently

You cannot trust a pre-registered acceptance table if the program doing the scoring is broken. The scorer is a program too, and it is wrong more often than the agent (read more about this dynamic in /blog/the-scorer-is-a-program-too-and-it-is-wrong-more-often-than-the-agent/).

This is why the instrument must be validated before the subject. Before my evaluation platform processes a sweep, it runs the scorer’s own self-test to validate the instrument before the subject. If the scorer fails its own self-test, the sweep halts before reading any agent outputs. You fix the scorer first.

Why we exclude infrastructure runs loudly

During batch evaluation sweeps, infrastructure runs occasionally occur. An environment timeout or network drop can abort a session before the agent ever speaks. These runs never produce a single agent line.

If you include these in your behavioral denominators, zero-line infrastructure aborts will artificially drag down your agent’s success metric. You will spend hours debugging a prompt that is actually working perfectly. However, if you silently drop them from the denominator, you invite data manipulation. You might accidentally drop runs that failed because the agent crashed mid-sentence, writing them off as infrastructure noise.

The solution is rigid categorization. The script inspects the event logs for every run. If no agent audio track was ever published, it is categorized as an infrastructure run. The evaluation script excludes it from the behavioral pass rate, but prints a highly visible warning with the exact count of excluded runs per arm. If repeated infrastructure runs survive filtering, the pipeline warns loudly so you can investigate harness health.