An AI agent that answers a returning customer usually starts from nothing. It pulls the order history, the past tickets, the notes, and reads all of it to work out who this customer is: what they reorder, what annoys them, what they tend to ask. Then it answers, and forgets. The next email from the same customer pays for the same reading again. A July 2026 technical report from the team behind Taobao's recommendation feed says this re-reading step alone accounted for about 95% of the compute in one stage of their pipeline, and describes the fix they shipped. Taobao is one of the largest shopping platforms in the world. The pattern behind the fix is small enough for a team of five to copy.
The result, stated plainly.
If your agent reads the same history about the same customer on every run, compile that history once into a small structured record, keep pointers from each line of the record back to the events it came from, and update the record against new events instead of re-reading everything. Taobao reports that this cut their user-modeling compute (the compute spent working out who each user is) to 44.20% of what re-reading cost: 33.43% for the reads, plus 10.77% to keep the records current. Between updates the record is out of date, and Taobao chose to recompile every two months. That interval is a decision to make on purpose.
The report covers three changes to the recommender, shipped together. The one this note is about carries over directly to an operations team: a memory layer the authors call the Memory Hub. Another, compressing the model's reasoning, is something you will meet through vendors rather than build, so it gets a short section. A separate finding, about fine-tuning a model on domain data, is the last part worth carrying. The rest (semantic item codes, a reward read from the production ranker) is recommender machinery, and we leave it out.
Your agent re-derives the same customer.
In the previous version of Taobao's system, a planning model read each user's full behavior history on every request before deciding what to show. For the most active users that history ran to about 55,000 tokens. The report says this planner "alone accounts for ~95% of the computational cost in agentic intent analysis," and that each pass "redundantly rediscovers patterns such as repurchase cycles, brand loyalty, and seasonal shifts that could instead persist and update across sessions."[1]
Now picture a Quebec industrial-supply distributor with an agent that drafts replies to customer emails. Before each draft, the agent reads the customer's two years of orders and tickets. On every run it concludes, again, that this customer reorders gloves every five or six weeks, gets upset when a Thursday delivery slips, and always wants the invoice copied to accounting. Those conclusions were true yesterday. They will be true tomorrow. The agent pays to reach them every single time, and the bill grows with the history, so the best customers cost the most to serve.
This is a statelessness problem. The agent has no memory of what it already understood, so it re-derives that understanding from raw material on each run. Our note on agent memory covered why memory became its own engineering discipline. This report adds a production cost to the question, and it shows what the fix cost to run. Our own knowledge base follows the same compile-once, keep-the-source pattern; our measurement of it covers the retrieval side, on a small sample of our own corpus.
The report's own sentence for this is the one to keep: the memory layer "converts a recurring per-inference expense into an amortized memory maintenance cost."[1] You stop paying per request to understand the customer. You pay now and then to keep the understanding current, and every request in between reads the result.
Compile it once, keep the pointers.
Taobao's record has four properties, and each one is a design choice you can copy.
Condensed into fixed fields. Each user's history becomes a short list of memory units. Each unit carries a pattern label drawn from a fixed list of hundreds of behavior types (extended only when nothing fits), a plain-language summary, preferred brands, when the pattern was active, and a timestamp.[1] The fields are fixed in advance. Free-form notes drift, and fixed fields can be checked.
Traceable to the source. Each unit stores pointers back to the events that produced it. The report checked both halves by hand: the pointers were correct 95.27% of the time across 21,268 annotations, and the pattern labels 82.89% across 2,514.[1] The pointers were right more often than the labels. A label or summary can be wrong while its evidence is right, and the pointer is how you get back to the evidence when it matters. The report gives no inter-annotator agreement for these checks. Pointers are also the defence when something false gets into the record: our note on memory poisoning covers why a remembered falsehood has nothing for a filter to catch.
Updated selectively. When new events arrive, a unit is rewritten only if the new events are relevant to it. Otherwise it is left alone, which the report calls "updating with evidence and retaining without disturbance." Events that match no unit become a new unit. One model call does both.[1]
Rewritten, not appended. This is the line most homemade memory systems miss. Each update "re-synthesizes the affected unit into a coherent snapshot of the user's current state rather than appending new behaviors to the old summary, preventing stale and emerging preferences from accumulating within one unit."[1] A summary that only grows ends up holding the customer of two years ago and the customer of today in the same paragraph, and the agent cannot tell which one is current. A separate study we covered, on AI that gets worse when the goal moves, measured the closest neighbouring failure: models get worse once the goal shifts, and restating the current goal only partly recovers it.
Here is what that looks like for the distributor, with one week of new events applied.
customer C-1187 · compiled 2026-08-01 · from 214 events
unit 1 pattern scheduled reorder
summary Reorders nitrile gloves and surface wipes every 5–6 weeks.
Same two products. Always by email.
sources orders 4471, 4502, 4538, 4590
unit 2 pattern delivery-sensitive
summary Escalated twice when a Thursday delivery slipped.
Both times a call settled it, not a credit.
sources tickets 912, 1040
unit 3 pattern invoicing
summary Wants every invoice copied to accounting.
sources tickets 877, 1102 | unit | new events this week | operation | record after the update |
|---|---|---|---|
| 1 · scheduled reorder | order 4631, four weeks after the last one | update | Rewritten: "Reorders every 4 weeks since July, previously 5–6." Order 4631 added to sources. |
| 2 · delivery-sensitive | none relevant | retain | Unchanged. Not touched, not re-summarized. |
| 3 · invoicing | none relevant | retain | Unchanged. |
| 4 · new | order 4640, first order of safety glasses | new | New unit: "Started buying eye protection in August." Source: order 4640. |
The price is a staleness window.
A compiled record is wrong between the moment something changes and the next update. Taobao says it plainly: "In production, incremental curation runs every two months."[1] For a feed serving hundreds of millions of people a day, a record up to two months old, read alongside the recent events, was worth the saving. The reads vastly outnumber the updates, so each update is spread across a very large number of requests.
Your interval will be different, and it should be chosen, not inherited from whenever someone set up the job. Three things set it. How fast does the thing you summarize change? Ordering habits move over months; an open complaint moves by the hour. How many reads does each update serve? If the agent reads a record twice between updates, the update is most of the cost and the saving is small. What does a stale read cost? A draft that misses a new ordering rhythm is a small miss. A draft that misses a complaint opened this morning is the one the customer remembers.
The report already does the one thing that shrinks the window most: the model reads the compiled memory together with the events since the last update, not the memory alone.[1] Copy that. The record covers the two years; the new events cover this week. You keep most of the saving, and the stale gap is limited to what the record says about events that have not been folded in yet. One step further is ours, not the report's: keep fast-changing facts, like an open ticket, out of the record entirely and read them live.
- How fast does it change? Habits and preferences go in the record. Open issues, balances and anything that changes by the hour get read live.
- How many reads per update? Count them. A record read twice between updates does not pay for itself; one read fifty times does.
- What does a stale read cost? Price the worst miss in the window. If it is a lost customer, shorten the interval or read the recent events live.
Fewer tokens is not the same as faster.
Another of the report's changes is aimed at the reasoning step. Taobao's model used to write out a long chain of reasoning before each answer. The team trained it to compress that reasoning into at most ten special tokens, which, the report says, can be decoded back into readable text when someone wants to inspect it.[1] You will not build this. You will meet it, though, because vendors are moving the same way, and the report's own measurement shows why a token count is not a speed claim.
For a buyer, the lesson is narrow. When a vendor says a new mode uses twenty times fewer tokens, ask what happened to the clock and to the bill, measured on your own task. A sharp cut in tokens can turn into a much smaller cut in time, because the remaining cost sits somewhere the token count does not show. The same goes for quality: in the report, compression alone held quality roughly level rather than improving it, and it took a later training stage to pass the written-out version.[1] Our note on reasoning effort as a cost lever covers how to price that setting on real work.
One more caution. The report says the compressed tokens "can be decoded back into a faithful, human-readable rationale whenever needed." It does not measure that faithfulness on held-out data.[1] A decoded explanation that reads well is not proof that it is the reasoning the model actually used. If a vendor offers "explanations on request," ask how they checked them.
Training on domain data alone collapsed their model.
The other finding worth carrying is a build-or-buy input. To specialize its model, the Taobao team kept training it on shopping data, mixed with about 10% general text (mathematics, code, science, medicine, instruction-following). Then they ran the same training with the general text removed, everything else held constant.[1] The model did not get a little worse at general tasks. It stopped working. This was continued pre-training on item data, with 65,536 new item tokens added to the vocabulary, in a single reported run. That is a harder setting than most "trained on your data" offers, so read it as a failure to test for, not a forecast.
Without the general mix, the model scored 0.12% on MMLU, 0.01% on CMMLU, 4.70% on GSM8K and 23.29% on IFEval. With the mix, the losses ran from 1.66% to 5.92%. The report's verdict: "General-domain data is therefore essential."[1] The instruction-tuning stage points the same way, though it was not tested the same way: the team kept roughly 20% general data there, a share it says was calibrated to preserve "producing well-formed JSON outputs and complying with complex multi-constraint prompts."[1] The team that built the model treated structured output as the capability worth protecting, and structured output is what every automation downstream of the model depends on.
For an operations team this lands in a sales conversation. Someone will offer a model "trained on your data." Ask what general data was mixed into that training, and how much. Then test it on your own prompts that require JSON and several constraints at once, before anything is wired to it. In most operations cases, a general model reading a well-kept compiled record should get the specialization without the risk. That is our read, not the report's, and nothing in the report tests it.
What to do this week.
- Find the re-derivation.
Log what your agent reads on each run for a week. If most of the input is the same history about the same customer, run after run, that is the cost this note is about. Our note on cost per step covers how to log it.
target: the share of each run's input that repeats from the last run - Fix the record's fields before you write a line.
A pattern label from a short list you control, a plain-language summary, the sources it came from, and the date it was last updated.
- Keep the pointers.
Every line of the record points to the orders, tickets or messages behind it. When a reply depends on the detail (a refund, a dispute), the agent follows the pointer and reads the source.
- Update by rewriting, not appending.
New events touch only the units they are relevant to. A touched unit is rewritten as today's picture of the customer. Unmatched events start a new unit. Everything else is left alone.
- Choose the window, then narrow it.
Set the update interval from how fast things change and how many reads each update serves. Have the agent read the record plus anything newer than it, and keep open issues out of the record entirely.
target: an interval written down, with the reason next to it - Test any "trained on your data" model for JSON first.
Ask what general data was mixed in. Run your own multi-constraint, JSON-output prompts against it before it touches a workflow.
What this does not claim.
Self-reported, unreplicated. Every efficacy figure in the report comes from the team that built and runs the system. No code, weights, codebook or evaluation data were released, and nobody outside has reproduced the results. The online test percentages come with no confidence intervals, and statistical significance is asserted for the traffic split without a test statistic. We treat the architecture as described and the numbers as Taobao's claims.
The memory layer is measured on compute only. The report also gives business results from its online tests. Those come from three changes shipped together, and no experiment isolates the memory layer's share of them. So we do not present any business result as the payoff of compiling memory. What the report measures for the memory layer is compute.
One figure is inconsistent. The report gives two different numbers for how much the memory layer shrinks the history (80% in one section, 94.5% in another) and never reconciles them. We rely on neither.
Their scale is not yours. A feed serving hundreds of millions of people a day reads each record a huge number of times between updates, which is what makes a two-month interval and a 10.77% curation cost work. A team whose agent reads each customer a few times a week will see a different split, and should count its own reads before expecting the saving.
Illustrative example. The distributor, customer C-1187, the orders, tickets and the weekly update are ours, built to show the three operations. They are not a measurement and not a client record.
If your agent reads the same history on every run and the bill is growing with your best customers, send us a sample of one run's input through the contact form. Working out what belongs in a compiled record, and what has to stay live, is the first step of our assistants and agents work.