Somewhere between a Tally gateway and a GSTR-2B filing is the data the frontier labs don't have.
the gap we work in
Every AI agent is scored by a piece of code called a grader. If that code can be satisfied without doing the real work, the agent learns to cheat — and every number you report afterwards is wrong. We attack your grader eight structured ways, close what gets through, and hand you a signed record of the whole thing that anyone can re-run.
The code that decides whether an AI agent did the job. The industry calls it a verifier; we use grader because it is the clearer word.
The agent scoring points without doing the work, because the grader is easier to satisfy than the task.
The thing we sell: a high score means the work was actually done. Not that a lucky string was printed.
Three steps, five working days, one record that outlives us. The panel below plays the whole loop on one of our own environments — pause it, replay it, or skip to the worked example. Every number in it comes from a file we publish.
You send a grader. The scoring code, one task it is meant to judge, and the sandbox you already have.
We attack it for five days. Eight structured attempts, each asking one question: can this be scored without doing the work?
You get the record. What broke, how each hole was closed, and a copy you re-run yourself in one command.
A grader arrives. It is the code that decides whether an agent’s work counts as done.
What you just watched is one of our own environments, run end to end. We are a new studio with no client work to show, so we publish our own instead of staging a case study. The full example, with every line of the record, is on the proof page.
Eight attacks against one grader. Four of them scored full marks without the task being done. Switch the fixes on one at a time and watch each attack stop working. The grader on the right rewrites itself as you go, so you can see which line did what.
# asserts ledger state, not strings def score(completion, answer, info): if info["fixture_id"] != info["run_id"]: return 0.0 led = info["ledger"].vouchers v = led.by_ref(answer.ref) if v is None or len(v.lines) != 3: return 0.0 return 1.0 if v.reconciles(answer, coerce=True) else 0.0 # negative suite: empty · garbage · our own wrong answer # all three must return 0.0 before this grader ships
Producing a training environment is now cheap, so volume is not the problem. The problem is that a grader which can be satisfied without doing the work does not merely fail to teach — it trains the model to cheat, and every score it touches stops meaning anything.
It pays the model to cheat. Here is what catching it costs.
of tasks sampled from a leading coding benchmark accepted a patch the maintainers verified as incorrect.
higher pass@1 on flagged-hackable tasks than on robust tasks of the same human-rated difficulty — across 134 model submissions.
for this long, a frontier lab froze changes to its RL environments, because reward hacks were outpacing review.
dollars, at the top of the range, paid per expert-authored environment task.
of the flagship coding benchmark’s tasks allow future-commit leakage — the agent can read the fix out of a later commit instead of writing one.
of the hardest unsolved problems on that benchmark had flawed test cases. OpenAI stopped reporting its score in early 2026 and moved to a harder set.
Same model, same task, same compute. The only variable is whether the grader can be satisfied without doing the work.
“Soundness matters most: high reward must mean the task was actually solved, not hacked.” Neolab researcher, Epoch AI, January 2026
The chart on this page is ours and it is a model, not a measurement: two runs of the same task, one with a grader that can be satisfied without doing the work. Every other figure is quoted from the source beside it, at the precision the source gives.
Everything arrives as one repository that you own outright. Publish it or keep it private — either is fine. We keep no copy after handover. See the four files in full.
Containerised replica of the target surface. Dependencies pinned, state deterministic, reproducible on a machine that isn't ours.
A reference trajectory that solves the task, so the environment validates without a model in the loop.
The grader. Programmatic wherever a check will do. Asserts state, not strings. Includes negative cases, not only positive ones.
Every attack we ran, which ones scored when they should not have, the reward spread across runs, and the diff that closed each hole. Signed and versioned.
The first two stages are mechanical and cost almost nothing, so they should run on every grade before a model ever trains against it — and almost never do. Only what survives them reaches a person. That gap is the whole business.
Does it compile, run, and reject garbage?
Known-good must pass. Empty, random and trivially wrong must fail. Mechanical, and free.
A frontier model is asked one question:
“Find a submission that scores without doing the task.” If it finds one, the environment fails.
Only survivors reach a person. What does a model not think to try?
Difficulty calibration, fixture bias, and the hole that sits in plain sight.
The three gates. The method is published, the funnel is cheap, and one good engineer can run it. It gets us in the door, it does not keep us there.
A growing library of attacks that actually broke something. Every environment adds its hacks and none are thrown away, so gate two gets sharper the more work we do.
$750 – 1,500 per environment is what a contracting firm spends, because a person builds and reviews every one. Here the expensive tier only ever sees survivors. Modelled, not billed.
Everything below can be opened right now, without talking to us. The last three rows are what does not exist yet, with the quarter each is due. A published date is the one promise a new studio can be held to.
| artefact | status | where / when |
|---|---|---|
| Probe library v0.1The eight attacks, what each one tests, and what closes it. Free to reuse and fork. | published | the method |
| Specimen audit — ledger-post-004One Tally task: the grader as submitted, the four attacks that beat it, and the exact change that closed each hole. | published | the worked example |
| Five-axis scorecardCompletion, functionality, generalisation, integrity, soundness — with the two that are still open marked open. | published | the worked example |
| Environment on a public hubInstallable in one command, so you can run our attacks against our own work. | in progress | Q1 2027 |
| Audit log for that environmentThe same treatment we sell, applied to something anyone can download. | in progress | Q1 2027 |
| Reproduction by somebody elseSomebody outside the studio re-runs one attack and publishes the result. The only row that proves the method travels. | not yet | Q1 2027 |
A big vendor ships hundreds of environments and reviews them by sampling. We ship few and audit every one.
One environment, one price. No retainer, no seat count, and no sales call needed to learn the number. The first audit is free, we reply once, and we do not chase you afterwards. If what you own is a judge rather than an environment, the same method runs as a judge validation; if what you own is the box the agent runs in, it runs as a containment audit.
Every lab partnership we can find so far is with Western software. The surfaces that get no attention are the business systems the rest of the world runs on, and they are where the training data simply does not exist. The five surfaces we start with, and why nobody has built them yet.
2.5M+ businesses. Desktop-first, XML gateway on port 9000, native JSON since TallyPrime R7.0.
GSTR-1, 2B reconciliation, session state that resists automation.
Books and Vyapar. The accounting stack of the Indian small business.
MCA21, EPFO, ClearTax. Surfaces with no Western equivalent.
Browser and desktop workflows anywhere. Ask — the method is domain-independent.
San Francisco can build a Tally environment. It is choosing not to.
Which one do you need?Prefer to just try it?
Send one grader and see what comes back. One page of findings, five working days, no call required.
Request a free auditOne grader per replyOne grader per reply · findings in 5 working days · no obligation, no follow-up sequence