An assistant builds the month-end margin sheet from the exports. The owner, sensibly nervous, runs it a second time, gets the same gross margin to the dollar, and signs off. Or a reviewer step reads the draft and the brief, says "looks right", and the quote goes out. Both checks share a weakness that a team at Google Cloud AI Research and the University of Cambridge has now measured on office work: neither one ever opened the files.[1] When the same model agrees with itself, that is one process repeating its reading of the task, and when a reviewer only has the draft, it can only judge how the draft reads. The paper's numbers put a size on both gaps.
If you want the practical part first, skip to the five steps.
Checking work when nobody has the answer key.
The paper's setting is the one a business is actually in. The same model does the work and checks the work. The checker never sees a grading rubric or a reference answer, because in real life there is none: if you had the right margin figure already, you would not be asking an AI to build the sheet. The tasks are long office jobs from five public benchmarks (1,166 tasks in all): rebuilding spreadsheets with formulas, producing documents and decks from a folder of inputs, and working through an occupation's work files with their own source data. Each task was run ten times to produce a pool of attempts (the paper calls them rollouts), each with its delivered files and its action trace, and every checking method was given the same ten.[1] The paper's term for the approach is agentic verification: the checker acts inside the task's files instead of reading the drafts.
Ten attempts per task is the paper's budget, not a suggestion for yours, and we come back to the bill below. The reason to read on anyway is that the authors broke the attempts into individual claims and measured two things that apply at any number of attempts: how often agreement among them was wrong, and how much a checker gained by going into the files instead of reading the drafts.
Unanimous is not the same as right.
The authors took the ten Claude Opus 4.8 attempts on each task of one benchmark (APEX-Agents, the one where that model scored lowest) and split the deliverables into claims, one per item the benchmark grades: 434 tasks, 1,692 claims. Then they sorted each claim by whether the ten attempts gave the same value.[1]
In the paper's claim-level sample (Claude Opus 4.8 on APEX-Agents, 1,692 claims), 34% of the values all ten attempts agreed on were wrong. Read the three bars together and voting fails twice. Where the attempts disagreed, the right answer was usually somewhere in the pool (74%), but picking the most common value got it less than half the time (47%). Where they all agreed, there was nothing to vote on, and a third of them were wrong. The paper's conclusion is the one to keep: a disagreement is a list of things to settle, and an agreement is a list of things to attack, because the attempts share a model, a reading of the task and a way of working, so they share mistakes.[1]
Scope it before you repeat it. This is one model on one benchmark, the hardest one for that model, and the same model did the claim extraction. We would not quote "a third of AI consensus is wrong" to anyone. What we would say is narrower and holds up: two runs that agree have told you about the model's consistency, not about the answer.
A reviewer that only reads the drafts buys a point.
The main experiment compares ways of getting a better deliverable out of the same ten attempts. Four of them only read: majority voting, a judge scoring all ten against the task in one call, a pairwise tournament, and a criteria-based verifier from an earlier paper that ranks candidates by comparing them item by item. Three go into the files: a plain tool-using checker with the same sandbox and no further instructions, and the authors' own harness (VeriHarness), once selecting the best attempt and once also revising it. Every method scored on each benchmark's own scale, averaged across the five, three checker runs each.[1]
The pattern survives rounding. Voting and a single judge over ten drafts moved the average by a point or less, on either model. The tournament and the criteria-based verifier did better, around two points, but a checker that was simply put in the sandbox with the files and told to score the attempts did about as well with no further instruction. The authors' harness, which goes into the files with a specific list of claims to test, added 4.4 and 4.1 points by selection alone and 6.2 and 6.4 once it also fixed what it found.[1] The shape matters more than the headline number: the methods that read were bunched at the bottom, and the only way up was to open the sources.
What the harness does with that access is three plain jobs. A disagreement resolver takes the claims the attempts split on, traces each candidate value back to where it came from, and runs whichever check would separate them, while allowing that every candidate may be wrong. A consensus challenger, in a separate conversation that cannot see the resolver, takes the claims all ten share, proposes how each could be wrong, and tests the proposal: recompute from the raw inputs, compare labels with the source metadata, reread the file the task actually names, compare the deliverable against the task for requirements nobody addressed. Then a fresh context reads the two evidence records, not the investigators' conversations, picks the attempt with the fewest defects the evidence supports, writes a revision plan, and ships the result with a record that links each claim to its check and verdict. Claims the evidence could not settle stay marked as unsettled.[1]
Most of the margin is in eighteen text files.
The ablations change what you would copy. The architecture above, run with an empty skill library, scored 51.3 and 53.8 (Flash, Opus) with revision. The plain tool-using checker with the same sandbox, no protocol and no skills, scored 50.7 and 53.1. So the resolver, challenger and fresh-context design is worth about 0.6 and 0.7 points on its own, inside the seed spread. Adding the skill library takes the harness to 53.4 and 56.1, another 2.1 and 2.3.[1] The two rows come from different tables over the same pools and the ablation table reports no spread, so treat the split as indicative. But the direction is clear: most of what separates the harness from "give the model the files and ask it to check" is eighteen short text files, some with scripts, each written for one kind of deliverable.
| kind | count | what it tells the checker | one example from the paper |
|---|---|---|---|
| evidence | 6 | how to read a file type as evidence, not as extracted text, with the scripts to do it | a workbook: formulas beside their cached values, and a diff between workbooks |
| resolve | 4 | what counts as deciding evidence when candidates disagree, and in what order of authority | a workbook: the input note cells first, then labels, then sibling formulas, then the task wording |
| falsify | 3 | the ways a whole pool of that deliverable type goes wrong together, one check per entry | a written answer: documents provided but never opened, governing clauses never cited, a statistic computed over the wrong set |
| repair | 5 | how the type is graded, how an unresolved claim is closed, what must never be edited | a code patch: edit the tree, never the patch text |
Two things about those files. First, the paper's own boundary is that the checker never sees the task's rubric, and that holds. But the five repair skills supply, in the authors' words, "how the type is graded", and the library was written from "general document-handling knowledge and development failures", on tasks that are part of the pools the main table is scored on. Part of the measured gain is therefore the checker knowing the graders' conventions for each deliverable type, learned partly from the test. That is not a flaw in the method so much as a description of it: a checker is as good as what it has been told about how this kind of work fails. Second, the "skills" bundle knowledge with tooling. Six of the eighteen are the scripts for reading spreadsheets, PDFs and documents properly, and the no-skills run loses those too, so the paper cannot separate "the checker could not read the workbook" from "the checker did not know what to look for".[1]
The checker stops where the work stopped.
The result we would put on the wall is about the challenger's failures, not its successes. The authors matched its verdicts on unanimous claims to the rubric items (Opus, APEX-Agents again). It did no harm: it never refuted a claim that every attempt had right. Then they sorted the shared errors it let through by why.[1]
- 70% it checked an input or an intermediate figure, which was correct; the error sat in a later step that is what the task actually graded
- 14% same figure, different definition, period or source version
- 12% it confirmed the wrong value
- 4% omission or mapping noise
Seventy percent of the misses were a matter of where it looked. The checker re-walked the path the work took, confirmed a step that was right, and did not go past the last step the attempts had taken. The paper's sentence for it: "The verifier tends to stop where the rollouts stopped."[1] There is a second, quieter version of the same finding. When the authors let the model rewrite its own skill library from development failures, two libraries evolved separately, one on spreadsheets and one on the APEX tasks, both arrived at the same new instruction: do not treat a recomputation that matches the pool as confirmation.
For a business deliverable: if the assistant built a subtotal and then a margin from it, a checker that recomputes the subtotal and finds it matches has confirmed the step the work showed. The client reads the margin. The check has to reach the figure that is delivered, traced back to the raw export, not the figure the draft built first.
What the gain cost.
The checking itself is cheap next to what it checks, and the reason is worth knowing: the harness works in short turns over one workspace, so most of its input is served from cache, while the read-the-drafts methods resend ten whole attempts on every call. The real bill is the pool. The whole method is roughly a tenfold generation spend for about six points, and the paper, which gives every baseline the same ten attempts to keep the comparison fair, does not answer whether to pay for the pool; the harness was not run at fewer than ten.[1][2]
The third tile matters most for a business, because it is the configuration you would actually run. Nobody at a small firm is going to build a verification runtime. You would paste the protocol into the instructions of a coding agent (the paper tested Claude Code, Gemini CLI and Codex). With Flash that matched the authors' own runtime within two points on every benchmark. With Opus it kept most of the gain on two benchmarks, about half on one, and fell back to a single attempt on two. The authors' explanation is that their runtime executes the protocol as a program (separate conversations, evidence passed as files), while a coding agent is asked to follow it as prose, and a step the prompt describes may or may not happen.[1]
How to build the checking step.
None of this says to run your invoices ten times. This is the quality-control step for AI work in a small business: how to brief the step that checks work an AI has already produced, and most of it costs instructions, not compute. These are our readings of the paper, labelled where they go past it.
- Give the checker the sources, not just the draft.
A review step that sees only the output can only judge how it reads. In this paper, voting or a judge over the drafts was worth a point or less, and the best read-only method about two. Mount the exports, the brief, the source documents, the previous version, read-only, beside the draft. If your checker cannot open the spreadsheet, it is a proofreader.
→ files beside the draft, read-only - Hand it a list of claims to test, not "check for errors".
Name the figures and statements in the deliverable the client will act on and make each one a claim with a source to trace it to. The paper's checker found its list in the disagreements between attempts; yours can come from the brief. Settle each claim against the source, allow that the draft's value may be wrong even when nothing contradicts it, and leave a claim open when the evidence does not settle it.
→ one claim, one source, one check - Attack what looks settled, with a falsify list per deliverable type.
For each kind of thing you produce (a quote, a reconciliation, a report section) keep a short list: the ways this type goes wrong all at once. Documents provided but never opened. A total over the wrong period. A label copied from the wrong column. The paper's evidence, resolve, falsify, repair split is a usable skeleton, and the resolve entry is an order of authority for conflicting sources: for us, signed scope, then written client instruction, then call notes.
→ a falsify list and an order of authority, per type - Check the figure that gets delivered, not the step the draft showed.
Seventy percent of the shared errors the paper's checker missed sat one step past an intermediate it had confirmed. Trace the delivered number back to the raw export in one unbroken chain. A recomputation that matches the draft is not a check unless it started from the source.
→ delivered figure to raw source, unbroken - Decide from the evidence record, in a fresh conversation, and ship what stayed unresolved.
Have the decision step read the checks and verdicts, not the investigator's reasoning, and start it clean so it carries no history of the draft (our note on reviewer labels has the other reason). Then keep the unsettled claims marked in what you send. The person who signs off reads that record (which claims were checked against which source, which stayed open), not the draft; that is our reading, not the paper's. A report that says which two figures could not be verified from the exports is a better report than one that silently chose.
→ fresh context, evidence only, unresolved items marked
One extrapolation, labelled as such. The resolver suggests a cheap trick at small numbers: for one high-stakes numeric deliverable, two or three independent drafts and a diff give you a short list of claims worth checking against the sources. The paper did not test pools smaller than ten, and with a single attempt every claim is a consensus claim, which fig 3 says is the harder case.
What this does not claim.
A preprint, days old when we read it, not rerun. The code, the dataset and the project site all resolve, which is more than most, but every number here is the authors' own.
The claim-level shares are one model on one benchmark. The 34%, 74% and 47% come from Opus on APEX-Agents, the benchmark where that model scored lowest, with the same model extracting the claims. The 70% is from the same slice. Neither is a general rate.
Most of the graders are AI judges, some from the checker's own family. Four of the five benchmarks are graded by a model: three wholly or partly by a Gemini model, one by a Claude Code agent on Opus 4.8, so each model being checked was, on some benchmarks, graded by its own family. In the paper's favour, the one benchmark graded by recalculating the workbook and comparing cells, with no model involved, shows gains in line with the rest (4.2 to 4.9 points). Several per-benchmark margins over the best baseline sit inside one seed standard deviation; the claim that survives cleanly is the average.
The architecture is not the lever; the knowledge is. By the paper's own ablations, most of the margin over a plain tool-using checker comes from eighteen hand-written skills that include how each deliverable type is graded, written with sight of development failures. Copy the brief, not the box diagram.
Ten attempts per task. Every gain in fig 2 was bought with a ten-attempt pool that the authors report cost over US$100,000 to produce across the study. Whether a smaller pool keeps the gain is not measured.
If an AI builds something your clients act on and the step that checks it only sees the draft, describe the deliverable and the check through the contact form. We will write back, free, with what the checker should be given and which figure it should trace. Building that step is part of our assistants and agents services.