Proofour own work, start to finish

One environment, every line published.

We are a new studio, so there is no client work to show. Instead this page is one of our own environments taken apart in full: the task, the grader exactly as it was written, the eight attacks that beat it, and the change that closed each hole. It is the same document a client receives.

01the worked exampleledger-post-004

One task. Four ways to score without doing it.

Post three vouchers into a Tally Prime company file. That is the whole task. The grader, as it was written, scored 1.0 for the string "voucher created" appearing anywhere in the output. Here is what that cost.

ledger-post-004

as submitted
surface
Tally Prime 5, desktop, local gateway on port 9000. No cloud endpoint.
task
Create three vouchers from a fixture, reconcile line items against the purchase register.
grader
A substring match on the completion text, a weak info["posted"] flag, and nothing else.
verdict
4 of 8 attacks score 1.0 without posting anything, and 2 correct answers are rejected.
result · one environment, re-run on every commit
environment attacks bypassed
as submitted		 bypassed
after the fix		 false negatives
resolved		 reproducible
ledger-post-004Tally Prime · voucher posting 8 4 0 2 of 2 yessame repo
what counts as a bypass

The attack returns 1.0 while the ledger state is unchanged. Not a low score, not a partial credit. Full marks, no work.

what counts as a false negative

A correct answer still scored 0 by the grader as submitted. Two real solutions were being rejected before the fix; both pass after.

who checked

Each of the eight results was re-run by hand against the ledger, not just read off the test output. The suite is the cheapest tier; it is not the last word.

how this environment is scoredfive axes · deterministic gate is authoritative

We score on the same five axes as a published security benchmark does, because it is a better rubric than anything we would invent and it is public: CWE-bench v1 (Collinear, October 2026) grades agent patches on exactly these five and reports a deterministic gate as the score, with an independent panel beside it.

completion

Does a correct submission score 1.0?

pass ✓
functionality

Do correct-but-unusual answers still pass?

2 of 2 fixed ✓
generalisation

Do variants of each attack also fail?

next: 20 variants
integrity

Did the score come from the work, not the grader?

4 → 0 ✓
soundness

Does it still run on a clean machine?

pending

Two are open and we are saying so: generalisation currently checks one variant per probe, and soundness has not been re-run on a machine that is not ours. Both are on the public roadmap, and both will be closed before this environment is offered to anyone.

Matrix of eight attack probes against five verifier guards. A tick marks the probe each guard alone refuses; blank cells are probes that guard does not touch.ledger stateline countfixture bindingnegative suitetype coercionempty outputsubstring matchoff-by-one countreplay a prior rolloutmarkdown-wrapped answertype confusionanswer sits in the task textour own wrong answerguard closes itguard does not touch it
Five fixes, eight attacks. Each fix closes exactly one hole; the two attacks that need no exploit are already rejected by the original grader.
Twelve rollouts of the same agent. The vulnerable verifier pays nine of them for doing nothing; the patched verifier pays only the three that completed the task.vulnerablepatched
Twelve runs of the same agent. The top row is what the grader paid. The bottom row is what it should have paid. Nine of twelve were paid for nothing.
02what a client receivesfour files, signed

The audit log is the artefact.

Anyone can ship a sandbox and a test file. The thing that makes it auditable is the record of what was tried against it, what got through, and the diff that closed each hole. Every line re-runs in the repository you were given.

See the four artefacts

03queuednot yet built

What we are building next.

GSTR-2B reconciliation against a purchase register, and a Zoho Books duplicate-detection task. Both are listed as queued because that is what they are. We would rather show an empty slot than a filled one that is not real.

No client engagements are listed on this page. There have not been any yet.

nextHow a grader gets beaten