Most of what goes wrong with an AI agent announces itself: an error, a stalled job, a surprise bill. This note is about the other kind: the silent failure, where nothing breaks and the answer is still wrong. In July, a team at Pasqal, a company that builds quantum computers, published a paper about putting an AI coding agent between the research literature and their own machines. In it they wrote down two times the agent got the science wrong while no run raised an error.[1] The jobs finished, the logs were clean, and the data or the explanation looked plausible. Only the physicists on the team, who knew what a right answer should look like, caught it.
A lab pointed its agent at its own machines.
The setup matters less than what went wrong, so briefly. The team built six skills for Claude Code, each owning one stage of the job: read a published paper or patent, turn it into an experiment plan, simulate the plan, simulate it again under the machine's current conditions, run it on a real quantum processor over the cloud, and analyze what came back.[1] One of the two processors sits in Sherbrooke, in Quebec's DistriQ quantum-innovation zone; the other is in Saudi Arabia. The paper reports three campaigns run end to end. You need no physics to follow the rest of this note, and we will not explain any.
The first campaign went the way a demo would. The agent reproduced a result first run on hardware in 2019, and "in less than an hour and three researcher prompts" had jobs running on the machine.[1] It recovered on its own from a paused processor and from failed jobs. The authors note that agents have the "healthy habit of consistently adding checkpoints, error logs" and error-catching code that keeps a run going.[1] Keep that habit in mind: a run that catches its own errors and carries on looks healthy, and in the next two campaigns, looking healthy was the problem.
Failure one: it measured the easy thing.
In the second campaign, the agent quietly measured what was easiest to compute instead of what would prove the result, and nothing raised an error. It was also the harder campaign. The source was a theory paper that predicted a particular pattern should appear under certain conditions, with no recipe for producing it on a real machine. The agent read the paper correctly and judged which parts were feasible on the hardware. Then, building the experiment, it narrowed. The source paper described four ways to measure whether the pattern had appeared. The agent kept two, and the authors say one of those could not tell the target pattern apart from look-alikes.[1] It also skipped a step of reasoning the result depends on.
The authors' own summary: the agent "consistently optimized for the nearest tractable sub-task at the expense of the core physics, selecting the observable that was easiest to compute rather than the one that genuinely certifies the phase."[1] In plain terms: it measured what was easy to measure instead of what would prove the result. "Because the pipeline ran smoothly and generated plausible-looking data, only domain expertise could identify that the wrong physical quantity was being measured."[1]
The business version is easy to picture (our example, not the paper's). An agent asked to reduce customer complaints reports tickets closed, because closing a ticket is the step it controls and can count. The dashboard is green. The customers are exactly as unhappy as before. Management has an old name for this pattern, Goodhart's law: when a measure becomes a target, it stops being a good measure. Our note on customer-service agent quality metrics covers the same gap: a satisfied rating is not a correct answer.
Failure two: it blamed the machine.
In the third campaign, the agent blamed the hardware for a problem caused by a mix-up in its own data handling. The campaign turned a patent into an optimization run over several rounds. Partway through, a high share of results came back breaking the rules of the problem. The agent "confidently attributed it to physics," diagnosing a specific kind of leakage on the processor.[1] It was a plausible story, told in the field's own vocabulary, and it was wrong.
The real cause sat in the agent's own plumbing. Two lists the workflow paired up, where each atom sat and what each measurement returned, were in different orders. Results were matched to the wrong labels, so valid answers looked invalid. In the paper's words, "the pathology lay in the workflow, neither the hardware nor the physics."[1] An expert found it by checking the raw mapping from results to labels by hand.
This one travels further than the first. Asked to explain something odd, the agent reached for a story about the outside system before it checked its own data handling. A confident outside explanation is worse than none, because it ends the investigation. If you have ever watched an automation blame a supplier's system for rejecting requests when the request had a date in the wrong format, you have seen the same shape (again our example, not the paper's).
Did it run, or did it do the right thing?
Put the two failures side by side and they share one structure. Neither raised an error: one produced plausible data, the other an odd result with a plausible, wrong explanation, and only someone who knew what the right answer looks like could tell. The authors' lesson from the second campaign fits in a sentence: agentic workflows "must be evaluated not merely on execution success, but on physical accuracy."[1]
- jobs submitted and completed on the real machine
- checkpoints and error logs in place
- recovered on its own from a paused processor and failed jobs
- data and charts looked plausible
- second campaign: measured the easiest quantity, not the one that proves the result
- third campaign: blamed the machine for a mix-up in its own data labels
For a business, swap "physical accuracy" for whatever right means in your process: the invoice matches the job, the quote uses this year's prices, the summary says what the client actually said. "Did it run" is the question your tools answer on their own: status pages, error alerts, completed tasks. "Did it do the right thing" has no automatic answer. Someone has to check it against something real: a sample of source records, a figure they know, the client's actual words. If no one is named to do that, no one does it, and the green dashboard becomes the only evidence anyone looks at. That named person is what "human in the loop" has to mean for a business: someone who checks the output, which is a different job from being able to stop the run.
3 messages, or 43: two case studies.
The paper also reports how much the researchers had to say to get each result right. The first campaign, reproducing a result already run on hardware, took 3 researcher messages. The second, where it quietly swapped in an easier goal, took 43 exchanges to reach a correct implementation.[1]
Read that pair with care. It is two case studies, the tasks differ in difficulty as well as in how well the agent understood them, and the paper counts "messages" in one place and "exchanges" in the other. It is not a measurement, and we would not quote a multiple from it. What it does show is the shape of the problem: in the campaign where the agent's idea of the goal drifted from the real one, getting to a correct result took far more back-and-forth, and the drift was silent. Nothing visible from outside the work could have told the team which of the two cases they were in.
That is the case for scoping a review step into an automation from day one, rather than adding one after something goes wrong. What you are budgeting for is not knowing in advance which case you are in. Our note on grading the brief before you blame the AI covers the input side of the same lesson: how precisely you state what done means sets most of what an agent can deliver.
What the structure caught, and what it could not.
Two design choices in the paper's workflow are worth copying in any business automation: one reviewable document holding the goal and settings, and a dry run before anything irreversible. Neither one caught the two failures, and the last paragraph of this section says why.
One document is the interface. Every stage reads and writes a single structured file, the experiment specification, holding the settings, what to measure and the limits. The paper calls it "the primary interface between researcher and agent: it can be inspected, modified or approved before anything is executed."[1] A change to the file flows to every later stage. The person reviews a document, not a chat transcript, and that is a far better thing to approve: it is short, it is the actual state of the work, and it can be compared with last week's version.
Nothing irreversible runs without a dry run first. No experiment reached the processor until it had been simulated twice: once under ideal conditions, and once with the machine's noise settings fetched live from the cloud service at the time of the run, not taken from a datasheet.[1] Comparing the two told the researcher, before any machine time was spent, whether the result would survive real conditions. The business version: before an agent sends, books, pays or deletes anything, run it against a copy, using today's data rather than last quarter's export.
Both safeguards were in place, and both failures got through anyway. The specification made the wrong quantity reviewable, but someone still had to know it was wrong. The dry run checked that the plan would execute, not that it measured the right thing. Good structure makes the human check cheaper; it does not replace it.
Who checks the agent? Four things to set before it runs.
Before an AI agent runs unattended, set four things: a written definition of right, a named person who checks the output, a rule to check your own data before blaming outside systems, and review time in the budget. These are our recommendations, built on the paper's account; the paper does not propose them in this form. None of them needs new software.
- Write down what right looks like, outside the chat.
Before the first run, put the success criterion in a short document the agent works from: the number that matters, the source of truth it gets checked against, and the threshold. If the goal lives only in a conversation, the agent can drift to the nearest thing it can measure, and you have nothing fixed to compare its output with.
→ one page: the goal, the measure, the source of truth - Name the person who answers the second question.
One named person who knows the subject checks the output against something real on a fixed schedule: say, ten records a week against the source documents. In a small business that person usually exists already: whoever did the job by hand before it was automated. Ideally not the person who built the automation, and not another AI model grading the first one; our note on AI judges grading their own homework covers why.
→ a name, a cadence, and what they compare against - When the agent explains a problem, check your own plumbing first.
Treat any confident explanation that blames an outside system (the supplier's system, the bank, the machine) as a hypothesis. Before acting on it, check the dull parts on your side: field order, units, which column feeds which field, date formats. In the paper, the bug was two lists in different orders.
- Budget the review before you know you need it.
Scope the checking time into the project at the start. If the automation lands in the three-message case, the time comes back to you. If it lands in the forty-three-exchange case, you find out before a client does.
A related failure, an agent that keeps reporting progress after it has stopped making any, is in our note on agents that stall but sound fine. For a fixed set of questions a checker can rerun, the note on a six-line eval suite sketches one shape.
What this does not claim.
One paper, one team, one vendor's stack. Nothing here has been independently replicated. Three of the four authors work at Pasqal, the workflow, as published, targets Pasqal's software and machines, and the fourth author is at Quantonation Ventures, a venture firm that invests in quantum technology.
Two failures are events, not rates. The paper does not say how often the agent substitutes an easier goal or misplaces the blame for a bug. It reports one of each: the first in its second campaign, the second in its third.
3 and 43 are two case studies. Different tasks, different difficulty, and the paper's own wording differs between "messages" and "exchanges." The pair illustrates a risk; it does not measure one.
We left out the paper's literature survey. The authors also used a second agent to sort several hundred research papers by whether they could run on today's hardware. Those labels were produced by an agent, and the paper reports no expert check of a sample, which is exactly the kind of output this note is about. We do not repeat its headline figure.
Models change. The study used Claude Code with Opus 4.8 and Fable 5 in mid-2026. The paper does not test whether other or newer models fail differently, or less often.
Not a verdict on any vendor or on quantum computing. The authors disclosed their own agent's failures, which is the reason the paper is worth reading. No client record is described, and we do not run this workflow.
Planning an automation and not sure who should check what it produces? Describe it through the contact form and you get a free written first read on where the review step belongs and what it should compare against. Agent work like this falls under our assistants and agents services.