The method is deliberately boring. List every way a model can score without doing the work, close them one at a time, and prove each closure with a test that stays in the repository.
We do not ask whether the grader returns a number. We ask what the model has to do to make that number one. Every fix below exists because an attack got through without it.
The original grader scored 1.0 when the string "voucher created" appeared in the model's output. Any agent that types those words wins. The guard opens the ledger and asserts the voucher exists, belongs to the right company, and has the right reference.
A two-line voucher scored the same as a three-line voucher. The guard asserts the line count, and reconciles each line against the purchase register rather than trusting the total.
Without a fixture binding, a rollout recorded yesterday passes today. The guard compares
fixture_id to run_id so a stale trajectory cannot be replayed into a fresh score.
Most graders are only ever run against a correct answer. We run empty output, garbage appended to a correct answer, and our own deliberately wrong answer through it before it ships. If any of the three scores above zero, the environment does not ship.
A correct answer returned as a markdown-wrapped string failed a strict comparison. The guard coerces before comparing, so a real solve is not rejected for formatting. False negatives are as expensive as false positives: they teach the model that being right does not pay.
This is the actual list, not a summary of it. Every environment we audit runs all eight, plus variant attacks built from the same pattern. Reuse it, fork it, run it against our work — the version number and date change whenever an attack is added, so you can tell whether you are reading the current one.
Two of the eight are not attacks: they are the floor and ceiling of the suite. An attack library that only contains exploits cannot tell you when your fix broke something that used to work. Probes 05 and 06 are the ones that catch false negatives — a correct answer wrongly rejected — which is the failure mode nobody advertises because it does not show up in a leaderboard.
Parsing, a known-good accept, a known-bad reject: fractions of a dollar. The expensive work is the model adversary and the human calibration pass, which is exactly why suppliers skip straight to shipping. That gap is the business.
Until this year, "your grader is unreliable" was our opinion. It is now a measured result with a protocol attached. We run that protocol, then we run the eight attacks on top of it, because the two answer different questions and a buyer needs both.
Reliability. Across 21 judges and roughly 541,000 judgments, exact-match agreement overstated chance-corrected kappa by 33 to 41 points, and judge rankings moved up to 14 places between benchmarks. A judge can be perfectly repeatable and still wrong.
Exploitability. Repeatability tells you the judge is consistent. It does not tell you whether an agent can satisfy it without doing the work. That is a different test, and it is the one we run.
Across 21 judges and roughly 541,000 judgments. A judge can agree with people on 90 out of 100 items and still be a third less right than it looks once you remove the agreement that easy labels hand it for free. Two production judges scored above 0.95 on test-retest while carrying position bias above 0.10 at the same time.
Published in June 2026, after a 21-judge audit. We did not invent it and we do not charge for inventing it. We run it to the letter, with confidence intervals, and then we run the attacks below it.
A note on honesty: partial adoption of this protocol is worse than none, because reporting kappa alone creates a false sense of having checked something. We report all five, or we say which one we could not run and why. If a client has no human-labelled calibration set, we say so up front and we do not sell a kappa we cannot compute.
The other half of this audit is containment — whether the agent stayed inside its own box while the judge was scoring it. That half is here, with thirteen vectors and the 2026 escape record.
One page. Kappa and position bias on a calibration set you already have. If the number is fine, you have lost nothing and you keep the page.
All five checks to the letter, the eight attacks, the audit log, and a paragraph stating what could not be validated and what we did instead.
Every judge upgrade is a new measurement. We re-run on each release, so the report you filed is still true twelve months later.
The formula is Cohen’s kappa: (agreement − agreement expected by chance) ÷ (1 − expected by chance). When most answers pass, chance already buys a high raw number, so the gap between the two is what your report has to show. A full report uses the confusion matrix rather than this two-slider version.
A task with no programmatic success criterion. If correctness is a matter of taste, we say so before an invoice exists rather than build something that cannot be falsified.
We also do not train policies. We build the environment and prove it is sound; what you train against it is yours.