The design is on every whiteboard. A cheap, fast model takes the first pass at each item. An expensive, slow model only steps in to check or to handle what the cheap one could not. Fewer expensive calls, lower bill, same quality. It is such an obvious win that almost nobody measures the whole run before building it. A 2026 preprint did measure it, in a setting where the pairing looked reasonable on paper, and the version most people would build came out slower than skipping the cheap stage entirely.
The result, stated plainly.
A cheap AI stage placed in front of an expensive one is not automatically cheaper or faster. In a 2026 Cornell preprint on speculative decoding, a small model drafting for a separately trained large one ran barely faster than the large model alone and slower under one common sampling setting. The same pairing turned into a clear speedup once the small model shared the large one's weights and cache (its learned parameters and its working memory of what it has already read). The outcome turned on whether the cheap stage started from the expensive stage's state or from scratch, not on how small it was.
The technique the paper measures is called speculative decoding, and the idea is simple enough to say in one sentence. A small model guesses the next several words, and the large model checks the whole guess in a single pass instead of producing each word itself; wherever the guess holds, the large model has saved work. It is the same shape as any cheap-first pipeline: a fast stage proposes, a slow stage verifies.
The paper's main subject is a way of training a family of model sizes as one nested model, so that each smaller size is a strict subset of the next. We are not going to explain that part, because no business reading this will train a language model. The result we care about is the control condition. The authors paired a small model and a large one trained the ordinary way, as independent models, and ran the draft-and-check loop. The draft was a fraction of the checker's size, although a larger fraction than the field usually uses, which is exactly what makes it look like a sensible production choice. Under greedy decoding (always take the likeliest next word) the pair barely beat the large model running alone. Under nucleus sampling (add some randomness, a common production default) it was slower.[1]
Then they ran the same loop with the nested pair, where the small model's layers (the stages of computation a model runs an input through) are literally the first layers of the large one, and the two share a cache. Same sizes, same task. The pairing became a clear speedup over the large model alone.[1] The authors' explanation is the part worth keeping: separately trained models develop different internal representations, so the draft and the checker disagree more often, and the draft's cache and compute never amortize against how often it is right. Shared weights and a shared cache remove both costs at once.
Three costs, one benefit.
A triage stage is any cheap step whose job is to decide whether, or how much of, an expensive step has to run. Strip the machine learning away and its economics are ordinary. A triage stage, any triage stage, has three costs. There is what it costs to run. There is what the next stage spends checking or redoing its work when it is wrong. And there is the price of starting cold: if the expensive stage has to re-read everything the cheap stage already read, the cheap stage has added a reading, not removed one. Against those three sits a single benefit, the expensive work the cheap stage lets you skip, multiplied by how often it is right. The pairing pays only when the benefit covers all three costs. Most whiteboard versions count the benefit and the first cost, and leave the other two out.
Here is the shape at a scale a small business in Quebec would recognize, the kind of document step that sits inside most AI agents run for business operations. A firm receives about 1,000 supplier documents a month, mostly emailed invoices and delivery notes, and wants the amounts, dates and references pulled into its accounting system. The expensive model reads each document once and extracts the fields. The cheap-first design has a small model do the extraction, then has the expensive model check it, and redo the ones the small model got wrong. Every figure below is illustrative, sized for this firm, and ours.
Walk the second bar. The cheap pass takes 25 minutes for the thousand documents. The expensive model then checks all thousand, and because it starts cold it has to read each document again to know whether the extraction is right: 75 minutes, three quarters of what a full extraction would have cost it. Three hundred documents fail the check and get redone in full, another 30 minutes. The cheap stage saved the expensive model a quarter of its reading on each document and charged for it three times: once to run, once to be checked, once to be redone. Set the cheap stage's accuracy to 90% and the bar still loses, because the from-scratch check alone is most of the expensive-only total. As long as the checker re-reads everything, no accuracy rescues this design.
The same rule, from the field.
One paper's control condition would be a thin reason for a field note. What lifts it is a second sighting of the same rule, from the opposite end of the stack. In July 2026 a practitioner channel described running the same code-review step twice on their own build, once on a cheap model and once on their strongest, as part of a multi-agent pipeline where many nodes checked their own work.[2] The cheap run returned a long list of findings. The strong run returned a short one. The long list looked more thorough until they read it: most of its findings flagged deliberate choices that the surrounding code explained, which the strong model had understood and the cheap one had not. In their words, "the cheap review hadn't saved us anything because now the review itself needed reviewing."
That is the triage rule again, with a different cost term carrying the loss. In the preprint, the cheap stage's overhead showed up as time: the draft's cache and compute, never amortized. In the pipeline, it showed up as cleanup: downstream agents spending tokens fixing what was never broken, and a person reading a list that should not have existed. One measured benchmark from one lab and one production post-mortem with no counts, pointing the same way: a cheap stage can be a net loss even when it is cheap, and you only find out by measuring the whole run. Our graph-engineering note drew the routing rule from that account, never downgrade the judge. This note is about the stage in front of the judge.
The lever is shared state, not a smaller model.
Return to what rescued the pairing in the preprint. The sizes did not change. What changed is that the small model became the first part of the large one, so the two agreed more often, and the checker inherited the draft's cache instead of rebuilding it.[1] The operator's version of that move has nothing to do with training. It is to make the expensive stage start from where the cheap stage left off.
In practice that means three things. Keep both stages inside one conversation or one context, so the second stage reads the first stage's notes rather than the raw document again. Pass the cheap stage's structured output and its reasons forward, not just its verdict, so the check is a confirmation and not a fresh read. And when the cheap stage was wrong, let the expensive stage continue from the draft instead of starting over. Our note on switching models mid-task covered a study in which handing a strong model a cheaper model's failed transcript cost it, and a clean restart did better. That study tested the escalation of a stuck attempt, not a verification pass, so we are extending it: handing a strong model a structured draft to confirm is a different job, and the cheaper read is the point.
The third bar is the same two models at the same accuracy. The confirmation now reads the cheap stage's extraction and the notes it wrote rather than re-reading the whole document, so it takes a third of the cold check. The 300 wrong ones are fixed by continuing from the draft rather than redone from zero. The pairing goes from 30 minutes slower than the expensive model alone to 35 minutes faster. Here is the same example priced, with illustrative per-document rates, since the time chart and the bill do not always move together.
| design | pipeline time | model cost | against the single stage |
|---|---|---|---|
| Expensive model only | 100 min | $30.00 | the baseline |
| Cheap first, cold start | 130 min | $32.00 | slower and dearer |
| Cheap first, shared state | 65 min | $13.50 | faster and cheaper |
The argument here is against a cheap stage that starts cold, not against cheap models. The effort setting and the model tier are both cost levers worth pulling, and both pull harder when the stages share context. A smaller model that re-reads everything is the design that loses in every column.
Three questions before a triage stage.
Before a cheap first pass, a pre-filter, a classifier that routes, a quick review that decides whether the slow review runs: the same three questions, in this order. The flow below is ours, drawn from the two accounts above.
The three questions as text, each with what a bad answer means:
- How often is it right? Measured on your own items. A triage stage that is right 70% of the time has to save more on the 70% than it costs on the 30%, plus its own run. If you do not know the rate, you cannot know whether it pays.
- What does it cost when it is wrong? Count the rework, and count who notices. A wrong pass that gets caught by the next stage costs a redo. A wrong pass that gets through costs more, and the bill lands on a person later. The cheap reviewer in the field account failed the second way.
- Does it start warm or cold? If the expensive stage has to re-read everything the cheap stage read, you have added a reading, not removed one. This is the question the preprint answers, and it is the one most designs never ask.
What to do this week.
- List every cheap-first stage you already run.
Pre-filters, classifiers that route, quick checks that gate a slow check, a small model drafting for a large one. Each is a triage stage, and each should answer the three questions.
- Time and bill the whole run, both ways, on 100 real items.
Expensive model alone, then cheap-first. Per-stage dashboards hide this: the cheap stage looks cheap on its own line. Our per-step cost trace is the instrument; the comparison is the number.
target: one end-to-end figure per design, time and cost, before anything ships - Make the second stage start warm.
Pass the first stage's output and its reasons forward. Keep both stages in one context where the platform allows it, and use prompt caching so the shared part is priced as shared. The confirmation should read notes, not the raw document.
- Never let the cheap stage be the judge.
A triage stage may propose. The stage that decides whether work is correct runs the strongest model you can afford, because its false positives are the most expensive output in the pipeline.
- Re-measure when anything changes.
A new model tier, a new document mix, a new price list: any of them can flip the ordering in fig 3. The rule does not change when the numbers do.
What this does not claim.
A preprint, one lab, one scale. The paper is a single-lab preprint on small research models; it reports one figure per condition, with no seed count and no error bars, and its training-cost figure is a throughput convention the authors themselves flag. We cite the direction of one control result and the authors' explanation for it, nothing more.
Token-level result, stage-level lesson. The paper measures a draft model guessing words inside one generation. We apply its cost structure to two separate model calls over a document. The costs sit in the same places; the paper does not test the second setting, and the warm-confirmation design in figs 2 and 3 is our extension. Neither source measured it.
Not about any named commercial model or tier. The paper measures small research models, not the tiers you buy through an API. The analogy to a cheap-tier model in front of an expensive-tier one is structural: the same costs, in the same places. Whether a given pairing pays is a measurement you make on your own pipeline.
Illustrative numbers. The firm, the documents, the minutes and the rates in figs 1 to 3 are ours, built to show where the costs sit. They are not a benchmark and not a client record.
One anecdote on the field side. The practitioner account has no counts and comes from a commercial channel. It is cited as an attributed instance of the failure mode, not as a measurement.
If you run a pipeline with a cheap stage in front of an expensive one and have never timed the two designs against each other, the contact form is the fastest way in. Tell us what the cheap stage decides and what happens when it is wrong. Measuring a cascade end to end before it ships is part of our assistants and agents work.