If you run LLMs in production, you almost certainly run an LLM judge somewhere: a model that scores another model's output and decides what passes. A June 2026 paper measured what that judge does when nobody splits its verdicts by author, and the answer should change how you run yours.
Everyone has a judge now.
A judge gates blog drafts before a human sees them. A judge scores support replies for tone and accuracy. A judge reviews generated code before it reaches the pull request. A judge ranks leads, grades summaries, filters outreach. The pattern spread because it works and it is cheap: one model call replaces a human review queue.
But it has a structural weakness that most teams never measure: the judge and the work it judges come from the same kind of system. And it turns out that matters, in a measurable direction.
Measured bias: 1.42× to 1.91×.
The Cambridge-led team built self-improving writing agents and used LLM reviewers to score their output against a benchmark of real accept/reject decisions on scientific papers. Their strongest conventional reviewer looked good on raw accuracy. Then they split its acceptance rate by who wrote the paper.
The reviewer accepted AI-generated papers at 1.42× to 1.91× the rate of human-written ones.[1] Same reviewer, same rubric, same benchmark. The bias has a name in the literature: self-preference, documented at NeurIPS 2024 under the title "LLM evaluators recognize and favor their own generations."[2] What the new paper adds is what the bias does inside a production loop, and a working correction.
Pause on what that means operationally. If your judge gates content, its pass rate on AI drafts is inflated relative to what a human reviewer would wave through. If your judge scores an agent whose prompts you keep tuning, the agent is learning to please a grader that already leans toward machine-flavored output. The loop optimizes toward the bias.
Open-ended judgment is the exposed surface.
The same paper ran a second judge in a different domain, and the contrast is the most useful design fact in it. Their math-proof grader was conditioned on a reference solution: it judged each proof against a known correct answer, step by step. That grader showed little self-preference and improved with no special correction. The paper reviewer had no reference answer. It judged each paper on its own terms, on plausibility and polish. That is where the bias lived.[1]
The practical split: reference-anchored judging is largely insulated; open-ended judging is exposed. "Does this output match the spec, the reference answer, the checklist" is a comparison. "Is this good" is a taste call, and an LLM's taste is trained on the same distribution that generated the work it is judging. Machine-flavored fluency reads as quality to a machine.
Most production judges are the exposed kind. "Rate this draft 1 to 10 for quality." "Would you approve this reply?" If your judge prompt has no reference artifact in it, assume the bias applies to you. (This is the same scorer-independence concern the frozen-environment note flagged from the RL side, now with a magnitude attached.)
Anchor, harvest, re-score.
The paper's correction has three parts, and all three translate below the research scale.
First, the anchor. The team never let a judge grade itself into a job. A judge was promoted only by beating the incumbent on a fixed, held-out set of ground-truth labels: real human accept/reject decisions that no judge could touch. Accuracy against the anchor, not vibes, decided which judge ran the next round.
Second, the adversarial pool. When a judge was replaced, the AI-generated papers the old judge had wrongly accepted were collected into an explicit pool, and the next judge was additionally rewarded for rejecting them, while still being held to accuracy on the anchor. In effect: your predecessor's false accepts become your regression test. The corrected reviewer gave up some raw benchmark accuracy and bought calibration with it: similar accept rates on human and machine text, 80% ground-truth accuracy retained.[1]
Third, the erasure. When the judge changed, every verdict the old judge had issued was discarded, and the affected work was re-scored under the new judge. The team ran a control that kept the stale scores. It never re-ranked anything: the old ordering stayed frozen and the new, stricter criterion never took effect. Mixed verdict histories do not blend. They pin you to the old judge's mistakes.
judge v1
while it runs
verdicts
this cycle
anchor set
ground truth
if it wins
adversarial pool
verdicts
never mix histories
Four moves for the judge you already run.
You do not need an evolutionary search loop to use any of this. The moves below are our translation to prompt-scale judging, the kind that lives in a system prompt and a cron job. The paper did not test at this scale; the logic transfers, the specific numbers do not.
- Build an anchor before you trust a verdict.
Twenty to fifty examples labeled by a human whose judgment you would defend, held out from everything. Measure the judge against it, and report two numbers, not one: accuracy, and the accept-rate split between human-written and AI-written items. The second number is the one nobody looks at, and it is where the 1.42× to 1.91× hides. An illustrative shape: a judge gating outbound drafts might show 84% anchor accuracy and a 1.6× machine-text tilt. The first number says ship it. The second says it is quietly lowering your bar.
target · accuracy AND author-split, reported together - Harvest false accepts into an adversarial set.
Every time you revise the judge prompt or swap its model, collect the items the old judge passed that a human later flagged. Rejecting those becomes part of the new judge's acceptance test, alongside the anchor. Expect calibration to cost a little raw accuracy. That trade is the fix working, not a regression.
target · old judge's false accepts = new judge's regression test - Version the judge like a model, because it is one.
A judge is prompt + model + rubric. Change any of the three and old verdicts stop being comparable with new ones. Do not average them in a dashboard, do not hold thresholds constant across the change, and if a decision depends on old scores, re-score under the current judge. The paper's control condition is the warning: mixed histories pinned the system to the displaced judge's mistakes permanently.
target · one judge version per verdict history, no mixing - Give the judge a reference whenever one exists.
Grade against the brief, the spec, the checklist, the approved example, not "is this good." Reference-anchored judging was the insulated case in the paper's data. Open-ended scoring is where self-preference lives, so reserve it for the decisions where nothing better exists, and put the human back in the loop there. (The CS-agent quality note's cross-family judge, Haiku judging Sonnet, attacks the same bias from the model axis; this move attacks it from the rubric axis.)
target · comparison over taste, wherever a reference exists
Sidebar: evaluation was the bill.
One more finding worth carrying out of the paper: in their system, generating work was cheap and judging it was expensive, roughly four dollars of every five. When they routed the high-volume calls through a small model and kept the expensive model where judgment concentrated, price-equivalent cost dropped about 13× at near-equal final quality.[1] Their numbers, their domain, one model pair. But the shape of the lesson is portable, and it is the same one the cost-tracing note argues from the instrumentation side: before optimizing your pipeline's cost, measure where evaluation spend actually concentrates. It is usually not where the dashboard points.
What this result is not.
The paper is a preliminary preprint and says so. Everything ran on one model at one setting, over short horizons, in intellectual-artifact domains (papers, proofs, code), with no human grading of the generated output. The 1.42× to 1.91× range is one reviewer on one benchmark, not a universal constant. Your judge's tilt could be smaller. It could be larger. That is precisely why move number one is to measure it rather than import a number from a paper, including this one.
And the correction is not free: the calibrated reviewer paid raw accuracy for its fairness. If your only metric is benchmark accuracy, a debiased judge will look like a step backward. Decide which failure costs you more: a judge that is slightly harsher on everything, or a judge that quietly holds machine output to a lower bar while your whole pipeline optimizes toward it.
If a judge gates something that matters in your stack and you want a second pair of eyes on its calibration, the contact form is the fastest way in. We do 30-minute reviews for production agent stacks, free.