Picture a business that has an AI draft its service pages and newsletters, then asks a second AI to check them before they go out: "Does this read well? Is it better than the old version?" The reviewer usually says yes. A September 2026 post from Meta's research lab suggests that yes means less than it sounds. Shown an expert human text and an AI text on the same task and asked which was better, strong AI judges picked the AI text 63.5% to 84.6% of the time. Given a rubric that another AI had written, they picked it every time.
What the judges got wrong.
The test task was writing one section of an academic paper. A model got a strong published paper with one section removed (the abstract, introduction, related work or conclusion) and had to write the missing section at about the original's length. The author's own section, held back, was the expert answer.
The authors then asked two strong models, Opus-4.8 and GPT-5.6-sol, to judge, in two of the most common ways. The first is the pairwise question: "which version is the better section? Reason step by step, then decide." Each pair was judged in both orders, with the texts shown as A and B, to cancel out any preference for whichever text comes first. Depending on which model wrote and which judged, the judge preferred the model's section over the expert's in 63.5% to 84.6% of cases. The second way is a rubric. GPT-5.6 wrote a prompt that generates a scoring rubric for each section, and under those rubrics the model's section scored higher 100% of the time.[1]
The authors explain why. The rubrics before learning were coverage checklists: name the paper's claims, methods and results. So they rewarded a "mini-paper" crammed into every section and marked the human original down for what it had deliberately left out.[1] An expert writing an introduction chooses what not to say, and a checklist cannot score a choice it has no line for. The same thing happens with a service page. A checklist that asks "does it mention every service, every guarantee, every city we cover?" will prefer the AI draft that lists everything over the page your best salesperson wrote, the one that leads with the one problem customers call about.
This is a different failure from the two we have written about before. In the label-bias study, telling the judge whose work it was moved the score; here the judge was asked only which of two versions, A or B, was the better section, and it still leaned toward the AI text. In the over-acceptance study, a reviewer accepted machine-written papers 1.42 to 1.91 times as often as human ones and was fixed by calibrating against adversarial examples. This post fixes the criteria instead, by learning them from expert writing.
What a better rubric rewards.
The authors' fix, which they call XAR (expert-aligned rubrics), turns rubric-writing into something you can measure. Take pieces of clearly above-average human writing and split each into a context and a continuation. Have the model write its own continuation from the same context. Each pair is a test: a good rubric should score the human piece higher. Then an optimizer model (Kimi-K2.6 in the main runs) gets to revise the prompt that generates the rubrics. It sees the current version, the size of the human-minus-model gap it produces and the cases where it got the order wrong. It is asked to widen the gap "for genuine-quality reasons, not superficial tells", in the authors' words. The prompt has a length cap, so the optimizer has to merge criteria rather than keep adding them, and the best version across rounds is kept.[1]
Their feasibility run was small: 52 paper-section examples from 13 papers, 8 papers to optimize on and 5 held back to check, and 7 rounds of revision. On the held-back papers, the gap started at −4.2 (the model's text scored well above the human's), crossed zero in round 4 and reached +2.76 (the caption puts the peak at round 5). It moved from both ends: the human texts' average score rose from 3.4 to 5.0 while the model's fell from 7.6 to 2.7.
What the rubric converged on is the part you can carry to your own work. Over the rounds, the optimizer rewrote the criteria that mis-scored expert prose: marking the human down for principled omissions, crediting a compressed summary as fluency, mistaking surface polish for craft. The rubrics it settled on reward a section that stays on its own job, careful selection, economy and precise detail that belongs to the section, rather than breadth.[1]
| criterion | the first rubric | the learned rubric |
|---|---|---|
| coverage | rewards naming every claim, method and result | rewards choosing what this section needs |
| omission | marks the human down for what it left out | credits a deliberate, principled omission |
| scope | rewards a miniature of the whole document | rewards a section that does its own job |
| detail | rewards the concrete specifics the checklist names | rewards precise detail on the section's own topic |
| polish | reads fluent restatement as quality | separates surface polish from craft |
The authors' examples of the weak model's writing match that list: an introduction that turns into a mini-paper, future work included; an abstract stuffed with numerical results it did not need; and, in fiction, stock similes such as "the air hung over the table like a warm cloth".[1]
Three things it needed.
The authors tested what the method depends on. Three of the conditions apply to anyone who uses an AI to judge writing, whether or not they ever train a model.
| condition | what the post reports | what it means for you |
|---|---|---|
| a strong enough judge | With Qwen3.5-27B as the judge, even the learned rubrics ranked the model above the human on stories (−0.55) and Wikipedia sections (−0.39); on paper sections, all four judges compared still favored the human. "A weak grader is incapable of seeing the gap." | A cheap, fast model as your reviewer may not be able to see the problem at all, whatever rubric you give it. |
| real expert examples | Against GPT-5.6, the learned rubric's gap was +1.39 for high-quality human papers, +0.95 for medium and +0.37 for low. | The test needs writing you would defend. The weaker the human sample, the smaller the gap; low-quality pages give it little to find. |
| matched length | Without a length target, models wrote sections two to three times the human length, and the extra length raised their rubric scores. The fix was a target of about ±15% of the human length. | Compare drafts of the same length, or the reviewer is partly counting words. |
The choice of optimizer mattered too: Kimi-K2.6 and Opus 5 found a positive gap and Muse Spark 1.1 struggled. That only matters if you try to learn a rubric yourself, which is the last option in the section on failing the test.
The headline claims, read carefully.
The authors then trained a 27-billion-parameter open model (Qwen3.5-27B) against the learned rubrics. The headline is that it ended up "ahead of every external frontier model" at paper sections, with a worst-of-three-rubrics score of 9.60 where the human original counts as 10.[1] Read it with three cautions.
The method is graded with its own rubrics. The evaluation rubrics come out of the same loop, and the model was trained against two of the three sets. The authors say so ("still potentially biased towards our model") and limit the damage by using a different judge for scoring and taking the worst of the three rubric scores. That is careful, but it is not an independent leaderboard, and we do not chart it as one.
The human checks are small, and they compare against the starting model, not frontier models. In blind side-by-side tests, the trained model was preferred 16 to 2 on paper sections (judged by the authors on their own papers) and 19 to 1 on story continuations, against the untrained 27B model. That is 18 and 20 judgments. The claim of beating frontier models rests on the rubric scores alone.
It did not work everywhere. On Wikipedia sections the trained model went from 2.7 to 4.0, still well below every frontier writer (GPT-5.6 at 7.9, Claude 5 Opus and Muse at 7.2). The authors attribute it in part to the judge used in training, which they found too weak a grader. And one learned rubric on its own could be gamed: after the first training round, the model scored 11.0 on the round-one rubric but 7.0 on the round-two rubric, below the untrained model's 10.4. (Scores above 10 mean above the human by that rubric's own count; the post does not reconcile them with its human = 10 scale.) The second round recovered it. A rubric your writer is tuned against stops measuring quality and starts measuring compliance with that rubric, which is why the authors keep re-learning it.
The known-good test, before you trust a reviewer.
You do not need the training half to use the judging half. The post's starting experiment doubles as a test anyone can run on their own AI review step: give it a pair where you already know the answer, and see whether it gets it right.
- Pick a page you would defend.
Your best page, email or post written by a person: the one that brought in work, or that a customer quoted back to you. The post's own data shows the gap shrinking as the human sample gets weaker, smallest for low-quality writing, so do not use a filler page.
→ a known-good answer - Have the AI write the same thing from the same brief.
Give the model the brief the human worked from, not the human text, and ask for the same length. If the draft comes back much longer, trim it or ask again: a longer draft wins on length alone.
→ a fair pair, same job, same length - Ask your reviewer, twice, blind.
Use the exact prompt your workflow uses. Run it once with the human page first and once with the AI draft first. Remove names, file names and anything that says who wrote which; a label alone moves an AI judge's score, as the label-bias note showed.
→ two verdicts, order cancelled out - Read the result, then repeat with a few more pairs.
If it prefers the AI draft both times, its "this reads well" is not evidence about quality. One pair is an anecdote: we would run several before trusting a pass. Redo the test whenever you change the reviewer's model or prompt.
→ you know what the reviewer's yes is worth
If your reviewer fails it.
A failed test usually means the AI review step is being asked the wrong question. Three moves, from cheapest to most work:
Give it questions it can answer. An AI reviewer is still useful for checks with a right answer: does every figure match the source, is every required field there, is anything on the banned-words list, is the length within range, does the French version say the same thing. Move the mechanical ones (length, required fields, banned words) into checks the software runs itself. "Is this good writing?" is the AI-opinion rung of the ladder from the runbook note, and on this evidence it is the weakest rung for copy.
Put the learned criteria in the prompt. If you keep a quality reviewer, stop asking it to confirm coverage. Ask it the questions from the table above: does this piece stay on its own job, did it leave out what it should, is every number needed, is there anything a reader could cut without losing meaning. Then run the known-good test again, because a better-sounding prompt is not proof of a better judge. One caution from us: the criteria in the table were reported for academic paper sections, in English. The post also learned rubrics for stories and Wikipedia sections but does not list their criteria, and nobody has shown any of them carry over to French copy or to sales pages, so test before you rely on them.
Learn a rubric on your own writing. The rubric loop runs without any model training. Collect pairs of your best human text and a model draft of the same brief, have a strong model revise a length-capped rubric prompt until the rubric scores the human text higher, and check it on pairs it has not seen. The result would be a reviewer tuned to your own standard. We have not run this, and nothing in the post shows it working at small-business scale or with a handful of examples; treat it as an experiment, not a practice. Keep a person as the final sign-off on anything a customer reads, either way.
What this does not claim.
It does not say AI writing is always worse, or that AI judges are always wrong. The finding is narrower: on one task, two strong judges ranked frontier-model writing above expert human writing, and a generated rubric made it worse. That is why the test uses a pair where you already know which text is better: a judge that gets a known answer wrong has not earned trust on the pairs where you do not know.
One blog post, no paper. Every figure is the authors' own run, reported on one page and not reproduced. The judge-failure rates come from one task (academic paper sections) with two models as writers and judges. The rubric-learning run used 13 papers. The post's charts were images we could not read, so we used only numbers stated in its text and captions.
The leaderboard is not independent. The "ahead of every frontier model" results are scored on the method's own learned rubrics, which the authors concede biases the ranking, and the blind human checks (18 and 20 judgments) compare the trained model only against its own starting point.
The training half is not advice. Acceleratech does not train writing models and does not recommend the authors' training setup. The part we use is the judging half: test the judge against a known-good answer first.
If you run an AI review step on your copy and want to know what its approval is worth, describe the workflow (no credentials or customer data) through the contact form. We will write back, free, with how we would set up the known-good test for it. Building review steps that check what they can actually check is part of our workflow automation services.