Picture an automation that drafts quotes, or matches supplier invoices to purchase orders, and gets one job in ten wrong. The vendor has just released a bigger model. Upgrading is one setting and a larger monthly bill, so it is the obvious move. An August 2026 paper suggests checking something else first. On one demanding benchmark, adding a better control layer around the same model raised its score by 9 points, against 1.8 for moving up a model generation (published reference scores, not a controlled rerun).
What the paper actually built.
The test bed is Terminal-Bench 2.1: 89 long, multi-step technical tasks that an AI agent works through at a command line, such as fitting a curve to lab data, processing video frames or editing DNA sequence files. The authors kept the model fixed and changed what surrounds it, the layer usually called the harness.
Their harness, StateM, has two parts: a runtime (the software that enforces the procedure) and a runbook: a plain text file (in YAML, a structured format) that splits a job into a few coarse phases such as plan, execute, review, hand off and repair. The agent moves through it with ordinary commands (where am I, what is next, go to this phase), and the software keeps the record of where the run stands outside the model's own working memory.
Two design choices do the work. First, every phase has a door on each side. Entering a phase loads that phase's instructions and the progress so far, so the agent is not relying on what it remembers from two hours earlier. Leaving requires the phase's exit checks to pass. If a check fails, the run stays where it is and the failure is written down instead of being carried into the next phase. Second, the runbook is an ordinary file that the agent and the person running it both read. The agent can propose a change; the person can edit it, keep versions and audit it.
The stop rule is the smallest piece and the easiest to borrow. When the agent tries to declare the job finished while the runbook says it is mid-way, a stop hook hands back the current phase and the obligations still open. The authors report one development run that kept going for 22 hours, with the stop hook and the run record kept outside the model holding its place. They also say a stop hook guarantees nothing on its own and needs time and retry limits. And they are clear about what the design is not: the protocol is "checked, logged, and recoverable", in their words, "rather than fully transactional".[1] If a step has already sent an email or changed a file, going back a phase does not undo it.
The numbers, each with its asterisk.
Set the title aside and here is the result the paper rests on. GPT-5.5, run under OpenAI's Codex agent, has a published reference score of 83.1% on the benchmark. With the runbook, the authors report 92.1%, with 88 of the 89 tasks solved at least once across five tries. For scale, under the reference setup, moving from GPT-5.5 to the next generation, GPT-5.6 Sol, moves the published score from 83.1% to 84.9%.
The other two results matter more to a small business, because they are about cheaper models. GPT-5.6 Luna, a lower-cost tier, went from 76.7% to 85.4% with the GPT-5.5 runbook applied unchanged, which the authors note is numerically above the 84.9% published for the pricier Sol. DeepSeek-V4 Flash, from a different provider, went from 82.7% to 88.1% once the runbook was re-adapted for it.
Read every row with its asterisk. The GPT "before" scores are published reference numbers, and the GPT-5.5 one came from a different version of the Codex agent; the authors say in a footnote that it is not a rerun, like-for-like comparison. The DeepSeek "before" is the authors' own run, which makes it the cleanest of the three. "Numerically above" is the authors' phrase for Luna, and it is the right one: 85.4 against 84.9 is half a point between their run and someone else's published number. All of it is one benchmark, with a runbook developed on that benchmark.
With those caveats in place, the direction is still useful. On long, multi-step jobs, much of what looks like the model failing is the process around it failing to keep track of where the job stands, enforce the steps, check the work and recover. A model upgrade is one lever, and on this benchmark it was the smaller one. A related field note reached a similar conclusion from a different study, on auditing an automation by reading one file: in a small controlled grid, changing the runtime moved scores more than changing the model.
What carries over to another model.
The transfer results are where a build-versus-buy decision comes in. The authors developed the runbook with GPT-5.5, froze it, and applied it to other models without changing a line.
Within the same model family, it carried. On GPT-5.6 Luna, zero changes produced the 8.7-point gain above. (It also carried to GPT-5.6 Sol, but that score is the title's unadjudicated submission, so we leave it out.) Across a provider boundary, it did not. Applied as is, the GPT runbook took DeepSeek-V4 Flash from 82.7% to 82.0%: a slight drop.
What did cross the provider boundary was the structure, not the specifics: the runtime, the shape of the runbook, the way tasks are routed to phases, the authors' short list of top-level rules (their "golden rules") and the loop for analysing failures. The concrete practices, the detailed instructions written from GPT's mistakes, had to be redone from DeepSeek's own. That re-adaptation cost US$37.02 in API charges. The final scoring run cost another US$15.20, and everything the authors spent on DeepSeek came to US$52.22.[1] Those are API charges only; the authors' own time is not in them.
- developed on GPT-5.5 whole runbook, as written · 83.1 → 92.1
- same family GPT-5.6 Luna whole runbook, frozen, zero changes · 76.7 → 85.4
- other provider DeepSeek-V4 Flash frozen runbook: 82.7 → 82.0, no gainkept: runtime, structure, routing, golden rules, failure loopre-adapted: the concrete practices, US$37.02 · → 88.1
For a business that runs automations on one vendor's model, that gives a working rule for a switch: keep the structure, budget to re-tune the details, and do not assume last year's instructions carry over. It also tells you what to ask whoever builds your automation: which parts are structure (the phases, the checks, the rules for changing them) that you should own because they survive a switch, and which parts are tuned to one model and will need redoing. The authors' re-tune cost under US$38 in API charges; in a small business, most of the cost is likely to be the hours someone spends reading the new model's failures. It is the procedure-side twin of what we saw with switching models mid-task, where context written for one model weighed on another.
Who checks the handoff.
The paper grades its own checks by how much evidence each one produces. The ladder is useful well beyond this runtime, because almost every automation has "make sure that..." lines in it somewhere.
| kind of check | what it gives you |
|---|---|
| the system runs it | A command or a condition the software evaluates itself: the file exists, the tests pass, the totals reconcile. Evidence you can reproduce, as long as the check itself is right. |
| a person checks | A human judgement. Strong, but it needs someone available at that step. |
| the agent attests | A checklist or message the agent fills in. Structured, but it is the agent's own word. |
| another AI reviews | An opinion about meaning. In the authors' words, it does "not constitute deterministic verification." |
Sort your automation's checks onto this ladder, then move the ones that matter up a rung. "Confirm the invoice total is correct" in a prompt is the agent's word. A rule that refuses the handoff unless the line items add up to the total is a check the system runs. An agent's own report that the work is done is not evidence that it is, which is the same lesson as an agent that stalls but sounds fine.
Three ways it learned the wrong lesson.
The most useful section of the paper reports the authors' own mistakes. Their runbook improves by studying failed runs and writing the lessons back in as practices. That process stored the wrong lesson in three distinct ways.
| wrong lesson | in the paper | in a small business |
|---|---|---|
| a guess stored as a rule | The video tasks did not say how precise the target frame had to be. The tuning picked default values by watching how the benchmark behaved, then stored them as if they applied everywhere. | A fix for one client's odd file format written into the procedure for every client. |
| the grader's habit learned as the task | In the DNA insertion tasks, the checker preferred the left-most valid insertion point, a convention absent from the task description. Repeated feedback can teach the runbook to follow it without anyone reading the checker's code. | An assistant that learns how one reviewer likes things, then applies it as company policy. |
| a real lesson at the wrong size | Too much compatibility procedure for a code-refactoring benchmark; too heavy a workflow for web tasks that were mostly lookups. | A ten-step checklist bolted onto a job that needed one check. |
Their rule is that experience must be filtered before it becomes memory, and that "effective procedural memory requires addition, revision, and removal." In one line: harness scaling is "an abstraction problem, not rule accumulation."[1]
Their second benchmark, a set of business tasks run on GPT-5.6 Luna with runbooks frozen before testing, shows the same thing from the results side. Averaged across six task families, the gain on unseen tasks was small: 0.55 of a point. In an exploratory subgroup of two families whose tasks shared explicit constraints, a running state and a clear finish line, the gain was 10.04 points. Two other families got worse, by 2.78 and 3.70 points, because the control sat on the wrong step. Thinner, better-targeted runbooks did better in later reruns, but those reruns came after the test results were known, so the authors report them as diagnostics rather than clean results, and on one family's unseen tasks the score did not move. One family was deliberately given no runbook at all, which the authors count as the right call. The same discipline applies to the instructions you already have: we went through ours in an audit of instructions written for older models.
Before your next model upgrade.
None of this needs a new runtime. These four steps work on whatever runs your automations today, from a no-code workflow tool to a scripted agent.
- Sort a month of failures into three gaps.
The paper names three. The agent lacked the knowledge at the moment it needed it (fix: give that step its own instructions when it starts). The procedure was known but not finished (fix: a check that blocks the handoff). A lesson from last week's failure was not used this week (fix: write it into the procedure, with a version). Count which gap your failures fall into before you price a bigger model. If nobody logs failures yet, start the log this week.
→ you know which lever you are paying for - Move each important check up a rung.
For every handoff that matters (a quote going to a customer, a payment going out), make the check something the system runs, or a person signs off, rather than the agent's own word.
→ "done" means a check passed - Write the procedure in one file, with a stop rule.
Phases, what each one needs on the way in, what must be true on the way out. Keep it where you and the agent both read it. Add one rule: the job is not finished until the last phase's checks pass, with a cap on retries and time so a job that cannot pass stops and calls a person instead of running up the bill.
→ a procedure you can read, version and audit - Switch providers by structure, then prune.
When you change vendors, keep the phases and checks, and budget to re-tune the detailed instructions against the new model's own mistakes. A cheaper provider is also a new place your data goes: if the automation handles personal information processed outside Quebec, Law 25 requires an assessment and a written agreement first (see where your data goes). Once a quarter, remove the rules that came from one odd case or one reviewer's preference.
→ the procedure gets thinner as it gets better
What this does not claim.
One benchmark, tuned on itself. The Terminal-Bench numbers measure the runtime plus a runbook adapted to Terminal-Bench, and the authors say the two cannot be separated. The business-task results are smaller in aggregate and come from unequal task families with a single run per arm. The paper's conclusion that "the model appears not to be the (main) bottleneck" is scoped by the authors to the settings they studied, and so is ours.
The comparisons are not controlled. Every GPT reference score is a published or public-submission number, not a rerun under the same conditions. The DeepSeek baseline is the only "before" the authors ran themselves.
The title's two numbers stay out. The 95.3% is a raw public submission that had not been adjudicated as of the paper's 11 August 2026 status date. The authors agree four of its rewarded runs should not count (94.4%) and report that scoring all nine flagged runs as zero gives 93.3%. The "$15 frontier run" pairs DeepSeek's score and cost with GPT figures drawn from two different evaluations, so it is not a like-for-like price comparison. The authors disclose both halves; we do not chart either.
Not reviewed, not tested by us. The authors are named but give no affiliation, and say the work was done on personal time. The paper says the code is open-sourced but gives no link. We have not run it, we do not recommend it, and Acceleratech does not build runtimes of this kind. What we take from it is the set of mechanisms: phase checks, a stop rule, a check ladder and pruning.
If one of your automations fails often enough that you are pricing a bigger model, describe what it does and where it goes wrong (no credentials or customer data) through the contact form. We will write back, free, with which of the three gaps we would look at first. Hardening workflows like this is part of our workflow automation services.