Picture an owner who asks an AI agent to write the test set for the automation that reads supplier invoices. The brief is one sentence: "write tests that check our invoice reader." The tests come back, the automation passes nearly all of them, and the project moves on. A September 2026 post from Meta's research lab suggests that pass rate says more about the sentence than about the automation. When the lab's agent built tests on its own, they came out so easy that a strong model scored 98 out of 100. A one-sentence statement of what to measure barely changed the result. A detailed specification plus a hand-picked list of sources did.
What the agent was asked to build.
A benchmark is a test set for AI: a fixed list of tasks, the material each task comes with, a known right answer and a grader. The authors asked an AI research agent to build one from start to finish and to keep improving it. Each round, the agent writes the tasks, the grader and its own reference answer. Solver models then attempt the tasks, and their scores tell the agent how hard the test is. An AI reviewer then checks the test itself, not the solvers. The agent reads all of that and revises.[1]
Three benchmarks came out of it, all meant to test other research agents. Graveyard Bench asks whether an agent avoids proposing research directions the record has already killed. SilentTrain Bench asks it to fix a training-code bug that lowers results without throwing an error. Rebuttal Bench asks whether an author's reply to a peer reviewer actually answers the reviewer's objection. In this setup a lower score means a harder benchmark, and the authors treat harder, with a passing review, as better.
The experiment varied one thing: how much a person told the agent before and during the work.
| setting | what the person supplies |
|---|---|
| no feedback | Nothing. The agent picks the topic, what to measure and how. |
| coarse | One sentence of intent, before the work starts. It fixes what to measure and leaves all of the how to the agent. |
| fine | A detailed task specification plus a curated list of source material, before the work starts. It fixes what to measure and much of how. |
| fine + during | The fine brief, plus a person who reads a round's results and writes what the agent should do next. |
What the brief was worth.
Unaided, the agent built a test that was nearly saturated: every solver scored 80 or more. At the round the loop selected, the in-loop solvers scored 90 (Muse Spark, the same model that built the test) and 84 (Muse Glimmer); Nemotron, a held-out model whose scores were never shown to the agent, scored 80, and Claude Opus-5, also held out, scored 98.0. The first three rounds of that run scored a perfect 100.0 on Muse Spark.[1]
One sentence of intent barely moved it, though that reading leans on the no-feedback baseline, which is a different benchmark (more on that below). On Rebuttal Bench, the coarse brief gave 84.4, 83.7 and 81 on the same three solvers: Muse Spark dropped 5.6 points (our subtraction), Glimmer moved 0.3 and Nemotron scored one point higher (a slightly easier test) than with no brief at all. On Graveyard Bench the sentence did help, by up to 13.8 points, but on a ten-task test that is about one or two tasks' worth of score (our arithmetic: each task is a tenth of the mean).
The detailed brief is where the change came from. On Rebuttal Bench the fine setting reached 43.5, 51.2 and 39.1, with every one of the authors' five review criteria passing. Claude Opus-5, which never fed back into the loop, went from 83.1 with the one-sentence brief to 65.9 with the detailed one.[1]
Read the comparison carefully.
The authors' summary is that detailed direction "halves the score of the solvers". Two things limit how far that goes.
The no-feedback baseline is a different test. Left alone, the agent chose its own topic, and the authors reuse that one run as the reference in every comparison. So 90 against 43.5 compares Rebuttal Bench with something else, and the halving claim inherits that. The comparison that holds is the one-sentence brief against the detailed one, on the same benchmark: Muse Spark 84.4 to 43.5, Glimmer 83.7 to 51.2, Nemotron 81 to 39.1 and Opus-5 83.1 to 65.9 on Rebuttal Bench; on Graveyard Bench, Opus-5 went from 90.7 to 84.4 and Nemotron from 75.4 to 52.[1] The drop holds in that comparison, though each setting is one run. Among the numbers the text gives, it is smallest on Claude Opus-5, the held-out model that scored highest throughout: 17.2 points on Rebuttal and 6.3 on Graveyard (our subtraction).
The detailed brief bundles two things. The fine setting gave the agent a detailed specification and a curated source list. The one-sentence setting got no sources. The post cannot say whether the spec or the sources did the work. For anyone briefing an agent, the practical answer is to supply both, because nobody has shown you can skip either.
That fits the July 2026 preprint we covered in an earlier note: the quality of the context an agent is handed explains a lot of what gets blamed on the model. Here what the agent produced was a test, and a sentence of intent was not enough context to make it a hard one.
When the loop stalls, change the how.
The SilentTrain run is the post's one test of a person giving direction mid-run, and it already had the detailed spec and sources. Through round 5 it sat above 92 on Muse Spark: the agent kept revising and the test stayed easy. The authors split the run at round 6. One branch carried on alone. The other got three instructions, all about how to build the test and none about what it measures: put the solver on a multi-hour job, build the environment from the complete source files instead of excerpts, and grade with behavioral tests against the original buggy behavior instead of a fixed answer checker.[1]
| solver | branch left alone | branch with three "how" instructions |
|---|---|---|
| Muse Spark | 88.1 | 64.2 |
| Muse Glimmer | 85.6 | 54.2 |
| Nemotron (held out) | 83.8 | 48.8 |
Read that with care: it is one split, and part of the gain may come from asking for a multi-hour job, which adds difficulty through length and compute rather than through a better question. The transferable part is the kind of instruction. When an agent loop you run stops improving (an agent rewriting a prompt or a rule set round after round with nothing moving), the first question is not "what should it measure?" It is how the work is set up: is the agent seeing an excerpt where it needs the whole file, and is the check a fixed answer key where it needs to test behavior?
Three guard rails for AI-written checks.
The second half of the post matters to anyone who lets an AI write a test, a rubric or a checklist and then works against it. The authors state the problem plainly: "A low solver score on its own is weak evidence, because a broken task, an unsatisfiable requirement, a mis-specified grader, and fabricated grounding all produce the same number."[1] An agent rewarded for making a test hard has an easy way out: make the tasks impossible. Their loop guards against it in ways we group as three gates and a fourth safeguard.
Gate A: the check's own right answer passes first. Before any solver runs, the harness runs the agent's reference answer against the real grader. The round is admitted only if that answer scores at least 0.9; otherwise it goes back for repair, up to ten tries.[1] It is the cheapest guard in the loop, and it closes the make-it-impossible exit. It is also the test-writing counterpart of the known-good test in our note on AI reviewers, which covered a post from the same lab: before a judge or a test judges anything, check it on a case where you already know the answer.
Gate B: something other than the score reviews the check. A reviewer reads the test itself on five criteria (construct validity, correctness, feasibility, usefulness and an overall verdict), and a round that fails any of them cannot become the base for the next one. The post names three defects that the score cannot see: answers leaking into files the solver can read, a topic too shallow or too well memorized to separate good solvers from weak ones, and source material the agent claims to use but never actually puts in the test. A leaked answer key looks exactly like an easy test, and missing sources fit any score at all.
| what happened | what the score said | what the review said |
|---|---|---|
| autonomous run, rounds 1–3 | 100.0 on Muse Spark, all three rounds | round 1 failed all five criteria and round 3 failed two; round 4 was the first to pass all five |
| detailed brief, Rebuttal Bench, round 5 | 47.4, the hardest version so far | failed feasibility, so it could not seed round 6; the loop went back to round 3 and reached 43.5 at round 8 with all five passing |
Choosing on the score alone, the authors note, "would have kept iterations the verifier rejects".[1]
Gate C: each revision starts from the best accepted version. The parent of each round is the lowest-scoring round that passed review so far, not the most recent one. A revision that the reviewer rejects, or that makes the test easier, does not become the starting point for the next. It is the same discipline as keeping a known-good version of a prompt and only replacing it when a candidate beats it on the checks.
The fourth safeguard: a solver that never feeds back. Nemotron ran after every round and Claude Opus-5 on the selected round, and neither result went back to the agent. The authors' stated purpose is catching a test that is only hard for the models it was tuned against. Nemotron's scores dropped with the detailed brief too (81 to 39.1 on Rebuttal Bench, 75.4 to 52 on Graveyard), so the difficulty was not only tuned to the in-loop solvers.
One caution the post does not raise itself: the reviewer behind gate B is the same model that built the test. Muse-Spark-1.1 proposes, solves, grades the answers and runs all five review criteria, so every "passed all five" is the model approving its own work. The held-out solvers check difficulty, not validity, and no human audit of the accepted tests is reported. Self-grading is a known weak point, and in your own loop the gate B reviewer should be a different model or a person.
Briefing your own agent.
Acceleratech does not build benchmarks for frontier models. Small businesses do hand agents jobs where the agent writes its own yardstick: test cases for an automation, a checklist for reviewing extracted invoice data, a rubric for grading draft replies to customers. Here is how we would apply the post to that kind of work. The steps are our reading, not the authors' advice.
- Write the spec, not the sentence.
"Write tests for our invoice reader" is the one-sentence brief. A spec says what the tests are for, which cases matter (credit notes, two-page invoices, a supplier who puts tax on a separate line), what a correct result contains field by field, what counts as partial credit and what the tests must not do. That is where the person's time goes, not into a review at the end.
→ the agent knows what to measure and much of how - Hand it the sources.
Give it a curated set of real material: twenty actual invoices with their correct entries, the three that broke the last version. Not "find examples". The post cannot separate the spec from the sources, so supply both. Strip personal and customer details first, or run the work where the data is allowed to go.
→ tests grounded in your cases, not generic ones - Make its own right answer pass first.
Before an agent-written test or rubric grades anything, run it against an answer you know is right: the correct entries for those twenty invoices. If the known-good answer fails, the test is broken. Fix the test before you change the automation. Then run one answer you know is wrong (an invoice with a swapped total). If the test passes it, the test is too easy. This second check is ours, not the post's.
→ gate A: a check that fails a right answer never grades anything - Review the check with something other than its score.
Ask four questions of the test itself: does it test what it says it tests, can a correct answer pass it, does it use the sources it claims to, and is the answer visible anywhere the system under test can read? Use a different model or a person, not the agent that wrote it.
→ gate B: a low pass rate is more likely to point at the automation than at the test - Keep the best accepted version, and change the how when it stalls.
When the agent revises the test, keep the version that last passed review as the base until a new one passes and does better. If revisions stop helping, look at the setup before the goal: whole files instead of excerpts, behavior checks instead of a fixed answer key.
→ gate C: no slide backward, and a way out of a plateau
The six-line eval suite we ship with agents is written by a person. These steps are for the case where an agent writes the checks.
What this does not claim.
One blog post, small runs. Ten tasks per benchmark, ten rounds, one attempt per solver per task per round, one run per setting, no variance, and one split for the stalled run. A swing of a few points on ten tasks can be one task. The post's curves are images we could not read, so we used only numbers stated in its text, table and captions.
The headline is cross-construct. "Halves the score" compares against a benchmark the agent chose for itself. Only the one-sentence against the detailed comparison is like for like, and it bundles the spec with the sources.
One model reviews its own work. The same model builds the tests, solves them, grades the answers and runs every review criterion. "Passed all five" is that model's own judgment, unaudited by a person.
Harder is not the same as better. The post scores a benchmark by a low solver score plus a passing review. For your own tests, a set your automation fails more often is not automatically a better set; a good test fails when the automation is wrong and passes when it is right, which is why gate A matters more than difficulty.
The authors expect the finding to move. "We note that as models get more powerful, we expect these trends to change."[1] They present the method as a way to track that, not as a fixed result. Their conclusion on the human side is the line worth keeping: deciding which problems are meaningful and worth making difficult "is a judgment we do not yet know how to delegate."
If an agent writes tests, checklists or rubrics in one of your workflows and you want to know what its pass rate is worth, describe the workflow (no credentials or customer data) through the contact form. We will write back, free, with the brief and the three gates we would put around it. Building agent workflows whose checks can be trusted is part of our workflow automation services.