When an AI step hands back a wrong answer, the instinct is to imagine it straining: a long, tangled run, visible effort, then a miss. A team with Hui Wei of UC Merced as first author, and colleagues at UC San Diego, UIUC and Adobe Research, looked inside small open models' reasoning and reports a different pattern.[1] On the runs that ended wrong, the model did less internal work at the steps that were supposed to be hard, the final check and the pulling-together of evidence. Failure looked like coasting. That is one paper's measured correlation on 3 to 4 billion parameter models, not a law, and the authors say so. It still changes what a careful reviewer should look for.
If you want the practical part first, skip to the four rules.
What the paper measured.
A reasoning model writes its work out step by step before it answers (chain of thought). The authors wanted to know how much internal effort each of those steps took. Their measure, which they call Step-Aware Reasoning Energy (SARE), follows one step's tokens as they pass up through the model's layers and asks how much the relationships between those tokens keep changing from one layer to the next. A step whose internal picture keeps reorganizing deep into the network is one the model is working on. A step that settles in the first few layers was resolved cheaply. Add up the change across all layers and you get that step's energy.[1]
They then grouped every step into seven clusters, unsupervised, from the shape of its final-layer representation, and read each cluster's role afterwards (setting up the problem, retrieving facts, comparing, synthesizing, answering), so energy could be read by phase. They ran three models on six standard benchmarks: two in math (GSM8K, MATH), two in commonsense (CSQA, StrategyQA, a true-or-false set) and two multi-hop question sets that require chaining facts (HotpotQA, MuSiQue), 800 questions per model and benchmark (687 for StrategyQA).[1] Two questions: does effort differ by phase and between right and wrong runs, and can it predict a wrong answer without knowing the right one?
One thing to hold onto before the findings: this measure needs the hidden states at every layer of the model. You can compute it on a model you run yourself. You cannot compute it on a model you reach through an API, because the API does not hand you its insides. That shapes everything in the practical section.
The wrong runs did less work at the hard step.
The first finding is that effort is not spread evenly. It clusters by phase, and the shape depends on the model and the kind of task. Mid-run fact retrieval sat at a steady, moderate level across models. The ends of the run held the extremes, but which end depended on the task.[1]
In math, Phi-4-mini spent the most on setup, translating a word problem into structure, and the least on the final step, which is often light once the work is done. In multi-hop and commonsense questions, Gemma-3-4B and LLaMA-3.2-3B often peaked at the final synthesis, where scattered evidence has to be combined into one answer.[1]
The second finding is the one worth remembering. When the authors split runs into those that ended right and those that ended wrong, the wrong ones tended to show lower energy at specific late junctions, most clearly at final verification and synthesis. In LLaMA-3.2-3B the two groups overlapped in early retrieval and began to separate in later steps where quantities were worked out and pieces combined.[1] The pattern comes from the authors' plots for three model and benchmark pairs, one per model; the paper reports no significance test for it. The model that failed was not thrashing. At the step that needed the most work, it did less.
The authors are careful about what that means, and so should we be: "the analysis does not establish whether lower energy precedes or contributes to failure, but rather that the two co-occur at specific reasoning junctions."[1] Low effort at the hard step is a marker that travels with wrong answers in these models. It is not shown to cause them.
A third, smaller result matters for anyone tempted to set a threshold: the models used very different ranges of this energy. Gemma-3-4B ran from about 10 to 27.5, Phi-4-mini from 6 to 18, LLaMA-3.2-3B from 2 to 7.[1] A raw value that is low for one model is high for another.
Four confidence scores went blind on true or false.
The second half of the paper asks whether these internal signals can flag a wrong answer after the fact, with no answer key. The authors turned each run into a short profile (seven statistics about the energy along the run, five about which phases the run visited and how often it switched or revisited them), added the run's token count, and trained a simple classifier to call each run likely right or likely wrong. They compared it with the run's length alone and with four scores that read the model's own output probabilities, the usual way to ask a model how sure it was: average log-probability, entropy, perplexity and a self-certainty score.[1]
On StrategyQA, the true-or-false benchmark, all four confidence scores scored an F1 (a 0-to-1 score that balances how many wrong runs a detector catches against how many false alarms it raises) of zero, on all three models.[1] The authors explain why: on a true-or-false answer the model's probabilities look about the same whether it is right or wrong, so tuning a threshold on them collapses to "everything is correct" and the detector never flags anything. The authors stress that this is not a bug in their implementation but a limit of output-level confidence on binary tasks.
Two cautions keep this result honest. The four scores tested were computed from token probabilities. A model that says "I am 90% sure" in words was not tested, and neither were large commercial models. And this is one benchmark: on the other five, the confidence scores posted nonzero numbers, though the whole-run signals matched or beat them on most cells.[1] The lesson is narrow and worth having anyway: on a yes-or-no output, a confidence score read from the model's own probabilities can be blind, and it will not tell you it is blind.
The free signal: run length, read with care.
On StrategyQA, the plain token count of the run, a number most model APIs already report, held an F1 of 0.46 to 0.58 where the four confidence scores fell to zero, and matched or beat the energy method on two of three models.[1] That number needs a reference point: on this benchmark, flagging every run as wrong would score about 0.58, 0.47 and 0.52 on the three models. Token count and the full method sit on that line, not above it; the confidence scores fell below it because their tuned threshold flagged nothing. The AUPRC view in the next section is where length shows real signal on this benchmark: above the chance line on two models, level with it on Phi-4-mini. Across the whole paper, token count beat the chance line in 12 of the 18 model and benchmark combinations and fell below it in six, three of them on Phi-4-mini. The paper treats length and energy as complementary and ships them together.
The next question is what the energy features add on top of length, since length is already inside the main method. On F1, our count from the paper's Tables 1 to 3: in 12 of the 18 cells the full method beat token count alone by less than 0.01, tied it, or lost to it. F1 struggles to separate the two, because nearly every F1 cell in the paper sits within 0.02 of what flagging every run would score; Gemma-3-4B on MATH is the clear exception. On the AUPRC measure the next section uses, the full method beat token count by 0.01 or more in 14 of 18 rows, by the widest margins on Phi-4-mini math (GSM8K 0.644 against 0.395, MATH 0.702 against 0.600).[1]
For a business on closed models, run length is the most useful signal in this paper: a closed model's hidden states are never visible, but the length of every run is. It is a modest signal, and the rules below treat it as one.
Reading the scores honestly.
Do not take the paper's highest numbers at face value. On the two multi-hop benchmarks the full method's F1 scores run from 0.89 to 0.97, which looks excellent until you read the appendix: only 5% to 17% of those runs ended right.[1] When most runs fail, "this one failed" is the safe call, and a detector that says it every time scores high: flagging every run as wrong scores an F1 of about 0.91 on HotpotQA and 0.96 to 0.97 on MuSiQue. A better view is the precision-recall area (AUPRC) the authors report in the appendix, set against how often runs actually failed, which is roughly what guessing would score.
| benchmark | model | runs that failed (rough chance line) | SARE + tokens | token count | best confidence score |
|---|---|---|---|---|---|
| GSM8K | LLaMA-3.2-3B | 0.35 | 0.479 | 0.420 | 0.286 |
| GSM8K | Gemma-3-4B | 0.16 | 0.217 | 0.199 | 0.210 |
| GSM8K | Phi-4-mini | 0.45 | 0.644 | 0.395 | 0.380 |
| MATH | LLaMA-3.2-3B | 0.66 | 0.760 | 0.728 | 0.585 |
| MATH | Gemma-3-4B | 0.51 | 0.838 | 0.779 | 0.600 |
| MATH | Phi-4-mini | 0.62 | 0.702 | 0.600 | 0.528 |
| CSQA | LLaMA-3.2-3B | 0.30 | 0.329 | 0.331 | 0.252 |
| CSQA | Gemma-3-4B | 0.30 | 0.500 | 0.454 | 0.374 |
| CSQA | Phi-4-mini | 0.30 | 0.372 | 0.428 | 0.260 |
| StrategyQA | LLaMA-3.2-3B | 0.41 | 0.382 | 0.451 | 0.000 |
| StrategyQA | Gemma-3-4B | 0.30 | 0.419 | 0.390 | 0.000 |
| StrategyQA | Phi-4-mini | 0.35 | 0.367 | 0.347 | 0.000 |
| HotpotQA | LLaMA-3.2-3B | 0.83 | 0.861 | 0.869 | 0.730 |
| HotpotQA | Gemma-3-4B | 0.83 | 0.837 | 0.813 | 0.751 |
| HotpotQA | Phi-4-mini | 0.84 | 0.894 | 0.882 | 0.770 |
| MuSiQue | LLaMA-3.2-3B | 0.92 | 0.921 | 0.900 | 0.881 |
| MuSiQue | Gemma-3-4B | 0.93 | 0.926 | 0.909 | 0.917 |
| MuSiQue | Phi-4-mini | 0.95 | 0.969 | 0.958 | 0.964 |
The chance column is our arithmetic: one minus the correct-run rate in the paper's Table 4. Highlighted token counts beat the full method. "Best confidence score" is the highest of the four probability-based scores in each row. The paper reports 0.000 for the confidence scores on StrategyQA; a ranking score cannot produce exactly zero on this measure when any run failed, so treat these tables as the paper computed them and the chance line as a rough guide, not an exact floor. The best confidence score also sits below the chance line in 14 of 18 rows, which a correctly oriented score should rarely do; the paper may have scored those baselines in the opposite direction. Read "beats the confidence scores" as weak evidence and "beats the chance line" as the test.
Read against the chance line, the picture is more modest and more useful. On MuSiQue, every method sits within a few hundredths of what guessing would score: there is little to detect when more than nine runs in ten fail. The real gaps are in math, where the energy method clears the chance line in all six rows, by the widest margins on Gemma-3-4B MATH (0.838 against 0.51) and Phi-4-mini GSM8K (0.644 against 0.45). Token count beats the full method in four rows of eighteen, and on LLaMA-3.2-3B StrategyQA the full method lands under the chance line while token count clears it.[1]
What to do with closed models.
You cannot measure SARE on the closed models most small-business AI workflow automation runs on. Two ideas survive that gap, and they turn into four rules that cost almost nothing. These are our readings of the paper, labelled where they go past it.
- Log the length of every run, per kind of task.
Token count was the cheapest signal in the paper that held up, and most model APIs already return it with each response. Record it for every run, grouped by task type, so you know what normal looks like for a statement reconciliation or a quote. The paper does not report which direction predicted failure in its data (length was one input to a classifier), so do not assume short is bad or long is bad. Flag runs that fall well outside the usual band for their task, in either direction, and send them to a person. One more caveat: the paper's runs were capped at 512 new tokens and sampled at temperature 0.7, so whether length carries the same signal in a closed model's longer or hidden reasoning is untested.
→ length per run, a normal band per task type - Find the hard step in the trace and check it was worked, not narrated.
The coasting finding has a cheap reviewer's version, and this is our extrapolation, not the paper's measurement: a suspiciously smooth run through the step that should have been hard (the check, the reconciliation, the combining of sources) is a reason to look, not a reassurance. In the trace or work log, "totals reconciled" with no line-level comparison, or "verified against the source" with no source opened, is the visible cousin of low effort at the verification step. Our note on checking AI work covers where that check has to reach.
→ the check step shows its work, or the run goes to review - Do not gate a yes-or-no decision on a confidence score.
All four probability-based scores went to zero on the true-or-false benchmark. If an automation auto-approves "no duplicate found" or "matches the purchase order" because the model's confidence cleared a bar, that bar may be blind on exactly this kind of output (our extrapolation: the paper tested probability scores on small models, not a stated confidence from a closed one). Gate yes-or-no answers on a check against the source, or on a sampled review, instead.
→ binary outputs: checked or sampled, never confidence-gated - Set thresholds per model, against that model's normal.
Energy ranges differed by a factor of several between models (fig 2). By our reading, the same goes for run length: switch models and last month's normal band is wrong. Re-baseline whenever the model or the prompt changes, or every flag you set is noise.
→ re-baseline on every model or prompt change
A worked example: statement reconciliation.
An illustrative Quebec bookkeeping practice runs an assistant that reconciles each client's monthly supplier statements against the ledger and writes a one-line result: "reconciled, no discrepancies" or a list of differences. Reviewing every one by hand defeats the point; reviewing none is how a missed credit note reaches a client. The paper suggests where to spend the review time.
| review trigger | what it reads | why, from the paper |
|---|---|---|
| run length out of band | token count against the usual range for a statement of that size | length beat the chance line in 12 of 18 cells, at no cost |
| check step narrated, not shown | the trace says "reconciled" with no line-by-line comparison | wrong runs did less work at verification (correlational, small models) |
| "no discrepancies" result | sampled review on a fixed share, never confidence-gated | probability-based confidence went blind on a binary answer |
None of these triggers needs a new tool: the token count is in the API response, the trace is already logged, and the sample is a rule in the workflow. What the paper adds is a reason to point them at the hard step rather than spreading attention evenly, and a warning not to trust a calm "all clear" on a yes-or-no answer. Whether these triggers catch more errors than a flat sample in this setting is untested; it is the first thing we would measure.
The authors' own next step goes further: use low effort at a high-stakes step to trigger a re-try at that step, or reward sustained effort during training.[1] Both are proposals, untested in the paper, and both need access to the model's insides. For closed models, they are not on the table.
What this does not claim.
Small open models only. All experiments use 3 to 4 billion parameter open-weight models. The authors flag generalization to larger models as an open question. Three architectures agreeing is the only cross-model evidence.
Correlation, not cause. Lower energy at the hard step co-occurs with wrong answers in these models. The paper does not show it causes them, and says so.
Offline, and inside the model. SARE needs the hidden states at every layer for every step, which the authors describe as best suited to offline analysis. It cannot be run on a closed API.
The high scores ride the base rate. Multi-hop F1 near 0.9 reflects how often those runs failed (83% to 95% of them), not detection power. Use the AUPRC view against the chance line.
Length alone often keeps up. Token count matches the full method within 0.01, or beats it, in 12 of 18 F1 cells and beats it in four of 18 AUPRC rows. The energy features add the most in math, by AUPRC, and only on models whose insides you can read.
A single-lab preprint, not rerun. Every number here is the authors' own. The phase profiles in fig 1 restate the paper's prose; we did not have the plots.
If an AI step in your business returns yes-or-no answers that nobody checks, or you are not sure which runs deserve a second look, describe the workflow through the contact form. We will write back, free, with which triggers we would put on it. Building review steps like these is part of our assistants and agents services.