Field notes
How AI agents and workflow automation behave once they run: worked examples, and a few of our own measurements. Each note says which it is.
- A one-sentence brief barely moved the AI agent. A spec and a source list did.
- We put a new monitoring tool through ten uses. Seven died, and the one worth billing died on the terms, not the tech.
- Your AI agent can notice anything. What it may do without asking is set per action.
- More candidates than budget? Score them before you spend, and rank for the goal you actually have.
62 notes
-
A one-sentence brief barely moved the AI agent. A spec and a source list did.
How much does the brief you give an AI agent matter? A September 2026 Meta FAIR research blog post by Kim, Jin, Li, Whitehouse, Weston, Yuan, Kulikov, Saha and Lanchantin (pre-paper, ten tasks per benchmark, single runs) had an agent build and revise its own tests. Unaided, they came out nearly saturated: Claude Opus-5 scored 98.0. A one-sentence statement of intent barely moved the result (84.4 on Rebuttal Bench for the agent's own model), while a detailed spec plus a curated source list took the same benchmark topic to 43.5, and Opus-5 from 83.1 to 65.9. Why the authors' "halves the score" compares against a different, agent-chosen test, why the post cannot tell the spec's effect from the sources', what three "how" instructions did to a stalled run, and the three guard rails for any check an AI writes for itself: its own known-good answer passes first, something other than the score reviews the check, and each revision starts from the best accepted version.
Harness
-
We put a new monitoring tool through ten uses. Seven died, and the one worth billing died on the terms, not the tech.
A new paid tool shows up every month, and the question is never whether it works. It is whether it earns a slot in the stack. In August 2026 we put one through our own ladder: Ahrefs' Firehose, a real-time feed of newly crawled web pages. The gap check came first, before the feature list, and found one genuine gap and one feature we already had for free. Ten uses were scored. Seven died. The one that would have paid, a brand-monitoring add-on for clients, died on the terms of service: client-serving use is not addressed in writing, a no-competing-product clause is live, and the whole product is a beta the vendor can discontinue without notice. The second-highest score died too, because a workflow we already run replicates it at no cost. What survived is one internal feed, and it sits behind a test that costs nothing. The ladder, the ten scores, the three clauses, the free-workflow comparison, and the gate that is still shut.
Ledger
-
Your AI agent can notice anything. What it may do without asking is set per action.
An AI agent that acts without being asked needs two settings, not one: what it may notice and propose, and what it may do without sign-off, set per action by how hard the action is to undo. Grant the first generously and the second one action at a time: a 2026 survey of proactive AI agents finds security, healthcare and law keeping consequential acts gated while converging on more initiative under bounded authority. The four checks before an unrequested act (is the need real, is it worth the interruption, is now the moment, can it be undone), why an agent's silence needs a daily heartbeat to be read, and a worked receivables example in which an illustrative acceptance rate rises from 39% to 77% once the checks are tuned. The result is a human in the loop placed action by action.
Lexicon
-
More candidates than budget? Score them before you spend, and rank for the goal you actually have.
Every operations lead has more candidate tasks than budget: automations to build, leads to chase, audits to run. An August 2026 preprint from Carnegie Mellon and Anysphere built an AI system that mines the mathematics literature for open problems, attempts every one, and hands a reviewer a short list. The mathematics is not the lesson. The lesson is how the system spends a budget it cannot stretch: a rough AI score recorded before any work was done turned out to be a usable triage signal, and the right way to rank the candidates changed with the goal. Rank by odds of success when you want the most wins; keep only the most important few, then rank by odds, when you want one big win. An illustrative worked example with twelve candidate automations and the budget for three, the three rankings side by side, and the three questions to answer before you rank anything.
Harness
-
Adding a cheap AI model in front of the expensive one made the run slower. Shared state is what fixed it.
Putting a cheap AI model in front of an expensive one is the standard way to cut an AI pipeline's cost. In an August 2026 preprint from one Cornell lab, a reasonable-looking pairing of a small draft model with a large checker, trained separately, came out barely faster than the large model alone and slower under a common sampling setting. The same pairing paid off once the small model shared the large one's weights and cache. A practitioner post-mortem from July 2026 reports the same rule from the field: a cheap review stage flooded a pipeline with false positives and, by the practitioners' own account, saved nothing, because the review itself then needed reviewing. Why a cheap stage's overhead has to amortize against how often it is right, an illustrative worked example at small-business scale in which the cheap-first design loses, and the three questions to ask before adding any triage stage: how often is it right, what does being wrong cost, does it start warm or cold.
Ledger
-
Your AI reviewer prefers AI writing. Test it on your best human page first.
Can you trust an AI reviewer that says your AI-drafted copy reads well? A September 2026 Meta FAIR research blog post by Weston, Kulikov and Saha (pre-paper, the authors' own runs) found that strong AI judges asked "which is better?" picked frontier-model prose over expert human prose 63.5% to 84.6% of the time, in both orders, and that a rubric written by an AI picked it 100% of the time. The cause: coverage checklists reward a text that says everything and mark the expert down for what it chose to leave out. A rubric learned from expert writing reversed it (the held-out gap went from −4.2 to +2.76), rewarding selection, economy and staying on scope, and it needed a strong judge, real expert examples and matched length. Why the post's "ahead of every frontier model" result is graded on its own rubrics, the known-good test to run on your own reviewer (your best human page against an AI rewrite of the same brief, same length, both orders), and what to do if it fails.
Harness
-
Should your AI agent ask, or just guess? Price the question first.
When an AI agent is missing an input, it can guess a default or stop and ask. Many builds pick one by reflex. A 2026 preprint gives a better rule: ask only when the answer is worth what the question costs, and judge that by what the finished work is checked on, not by how unsure the agent feels. A worked example with illustrative numbers, the three checks and the table we write into an agent's instructions, and why the model will not find the efficient questions on its own.
Lexicon
-
You're billed by the token. A token is not a fixed amount of text.
Your AI bill, context window and document chunks are all counted in tokens, and a token is set by the model's tokenizer, a separate program its maker trained once. The same page can cost a different number of tokens by language and by model. What that changes in a budget and a vendor comparison, why French is the open question, and how to measure your own ratio instead of trusting a rule of thumb.
Ledger
-
The AI tool passed the safety check. We still said no.
In August we gave two questions that separate a trustworthy agent-setup prompt from a dangerous one. On 10 October 2026 we evaluated REA, a popular, well-engineered open-source reverse-engineering add-on for coding agents (version 6.3.0), and it passes both: a dry-run plan, approval before any write, a skill that tells the agent to ask first. We did not adopt it. Install hygiene is one gate of six. The other five, business-lane fit, host fit, release velocity and pinning, global footprint, and the legality and data path of the use case, are the ones no install guide asks you to run, and REA cleared only one of them cleanly for us. The six-gate table against the August case, what a reasoned no looks like, and how to run the gates on the next AI add-on someone asks you to install.
Lexicon
-
Claude skills that behave: fix the file layout before the wording.
Anthropic's skill authoring guide is mostly about how a skill's files are laid out and what checks it runs, not how its instructions are worded. We audited our own 82 skills against its checklist on 1 October 2026: one SKILL.md over the 500-line budget, one dead link, 35 with Windows paths, 18 long reference files without a contents list, now checked by a hook. What to fix, what to skip, and the rule we mostly skipped: evaluations first.
Lexicon
-
Your AI instructions have a shelf life. Audit them before they fail.
Anthropic's Claude 5 prompting guides say instructions written for older models can degrade the output of newer ones, and one kind of line can make a request fail outright. We audited our own 163 instruction files against the guides: zero reasoning-echo lines, about 760 hard rules, roughly half without a stated reason. What to delete this week, what to keep and explain, and where the vendor's own guides disagree.
Lexicon
-
When your AI attempts agree, that is not a check. Send the checker into the files.
An October 2026 preprint from Google Cloud AI Research and Cambridge ran two AI models ten times on each of 1,166 workspace tasks (spreadsheets, documents, work files) and then tried to check the results with no answer key. Voting or a single judge over the ten drafts added a point or less over one attempt. A checker sent into the task's files to test specific claims added 4 to 6. On the benchmark where that model scored lowest, a third of the values all ten attempts agreed on were wrong, and 70% of the shared errors the checker missed sat downstream of a step it had confirmed. What a small business keeps: give the checker the sources, treat agreement as a list of things to attack, and check the figure that gets delivered. The gain was bought with ten full attempts per task.
Harness
-
Never tell an AI reviewer whose work it is. The label alone moves the score.
A KAIST preprint, first posted in June 2026 and revised in September, had ten AI models score identical work twice, once labelled "your own selection" and once "another language model", with no model named. The label alone moved the average score by 0.43 on a 7-point scale, in both directions, and a false label moved it as much as a true one. Blind, with quality and judge severity controlled, self-preference fell to 0.003 among the eight judges that used the scale (0.18 across all ten). What that means for any AI review step in a business, the one-sentence fix, and a ten-minute test you can run on your own review prompt.
Harness
-
AI agents will use your website or route around it. WebMCP bets on the first.
WebMCP is a proposed web standard that lets a website register tools for AI agents: a name, a description, a JSON Schema, and a callback into code the site already runs. Agents act through the page instead of guessing at buttons or routing around it through a backend integration. An operator's explainer: the disintermediation stake, the pattern, the context budget every tool spends, and why neither first-party source measures the benefit.
Lexicon
-
Can you audit your AI automation by reading one file? Build it so you can.
An August 2026 position paper argues for running one unmodified AI agent runtime behind every automation, so that what each job does lives in a plain-language instructions file and checking it means reading that file. What a small business keeps from the design, why what an agent may reach belongs in its access grants rather than its prompt, and what the evidence says about runtime versus model: in a small controlled grid running from no checking loop to a full one, changing the runtime moved scores 8.5–13 points and changing the model 2.5–5; across three finished agent tools, one vendor reports its model stayed within 4%.
Relay
-
The AI agent's run was healthy. The answer was wrong. Only someone who knew the subject could tell.
A quantum-hardware company published two times its own AI agent got the science wrong: it quietly measured the easy quantity instead of the right one, and it blamed the machine for a bug in its own data handling. Neither run raised an error. What that means for any business automation: "did it run" and "did it do the right thing" are different questions, and only the first answers itself. In two case studies from the paper, the researchers needed 3 messages in one and 43 exchanges in the other: an illustration of the risk, not a measurement.
Harness
-
Your AI is stuck. Restart the stronger model, don't hand it the transcript.
Switching AI models mid-task has a measurable handoff tax. In a 58,000-run study of coding agents, handing the stronger model the cheaper model's full transcript recovered under half the quality gap, and in one model family a clean restart was both cheaper and better. The direction flips going down: a cheap model does well with the strong model's notes and worse without them. Two rules: escalate with a fresh brief, downshift with the notes attached.
Ledger
-
Running AI on your own hardware loses on cost. Buy it for compliance, or not at all.
We priced a 24 GB local-inference box against rented GPU time in September 2026: the box has to replace 4,700 to 23,600 rented hours before owning breaks even, on compute alone. So for working-hours use, the only real reason to own the hardware is compliance, which for a Quebec business means Law 25 and confidential client material. Why storage residency is not processing residency, why VRAM capacity decides whether a model runs at all, and the order to test in.
Ledger
-
An AI that remembers can be lied to once and keep the lie. A filter will not catch it.
A study of agent memory poisoning found that plain false statements, 1.2% of an AI agent's memory, cut its correct answers from 85% to 30%, and a write-time screen that catches most prompt injection let every one of them through. Why a false fact has nothing for a filter to find, the trust-weighting trap in the read path, and the three-line memory hygiene posture for a small business: control what can write, record where every memory came from, audit the store.
Sieve
-
The agent got a faster tool and finished fewer tasks. Routing is the hard part.
The Qwen team's technical report on Qwen-CUA, an AI agent that runs software from screenshots with only a mouse and keyboard, holds a result worth knowing before you automate: given a command-line tool as well, the agent took fewer steps and finished fewer tasks. What the team measured, why pixels are the fallback rather than the fast path, and the prompt-injection table that shows why the agent you deploy matters.
Lexicon
-
Autonomy is earned. This is the order you earn it in.
Build AI agents in this order: simplest primitive, harness, context, eval gate, the right loop, cost and termination as primitives, then a bought orchestration layer. Autonomy is what sits on top of the seven, not the first thing you build. Seven decisions that separate an agent demo from something you can leave running, with every outside source named at its real confidence tier.
Lexicon
-
n8n is source-available, not open source. Read the clause before you build on it.
n8n, the workflow automation platform, calls itself source-available, not open source: you can read every line of its code, and two sentences in its Sustainable Use License decide what you may do with it commercially. The clause in full, n8n's own lists of what it allows and rules out, three easy-to-miss mechanics in the same file (the .ee. enterprise carve-out, the master-only branch rule, a single 30-day cure), and what self-hosting, n8n Cloud or a provider running it for you means for your license.
Relay
-
An AI agent can run your ad account. Keep its reads away from its writes.
A marketing agency's co-founder says an AI agent has run his Facebook ad accounts for four months: making the ads, publishing them, pausing the losers, promoting the winners. Set aside the sales pitch and the results he reports, and one design decision carries the system: the agent changes the account through the ad platform's API and reads its numbers from a separate data warehouse. Why that split matters, how to keep the test-and-promote loop inside walls, and why the agent ends up making the same ad over and over.
Lexicon
-
We measured our own AI second brain. The index is the product.
An LLM-maintained knowledge base only pays if the AI can find the right page cheaply. We benchmarked ours on six questions: scoring a one-line index before opening any file matched free search on correctness while capping the worst case, 13 tool calls down to 7 and none of the six above 26k marginal tokens. Small sample, our own corpus, measured by us. The win is variance, not accuracy, and the index lines have to be written in the words a future question will use.
Probe
-
The agent kept writing status reports. It had stopped working.
Sakana AI put seven models in charge of a coffee roasting business for 90 simulated days and scored them on money earned. Most turned a profit. One sat idle 40 days out of 90 while still writing coherent status reports, a failure mode the authors name idle-drift. What separated the winners was behavioural, not intellectual: they contacted counterparties and they closed. Why action rate and messages sent belong in your agent monitoring next to did-it-finish.
Harness
-
The effort setting on your AI is a cost lever. Treat it like one.
Light, medium, high, max: almost every major AI model now has a reasoning-effort setting, and each step up buys longer thinking at a higher bill. Sebastian Raschka's July 2026 survey of six open-weight technical reports shows how the setting is trained in, why the gains flatten near the top, and why a smaller model at high effort can sometimes match a larger one at low effort. What each setting buys, which one to start with, and how to price it on your own work.
Ledger
-
Before you blame the AI, grade the brief you gave it.
When an AI agent drifts, invents facts or misuses a tool, the model takes the blame. A July 2026 preprint grades the context the agent was handed instead, on seven criteria, before any behavior test. Structure did most of the work, adding rules on top slightly lowered the behavior score, and the cheapest brief was the least reliable. The seven criteria as a self-check for your own agent, with the caveats a single-author study needs.
Harness
-
Before you pay for a bigger AI model, fix the runbook around the one you have.
Is a bigger AI model the fix when an automation keeps getting jobs wrong? An August 2026 preprint by Qin, Lu, Wang and Wang (arXiv 2608.15089) put a state-machine runbook around the same model on Terminal-Bench 2.1: GPT-5.5 went from a published 83.1% to 92.1%, while the next model generation moved the published score 1.8 points. The cheaper GPT-5.6 Luna went from 76.7% to 85.4% with the runbook unchanged, and DeepSeek-V4 Flash from 82.7% to 88.1% after US$37.02 of re-adaptation. A tuned runbook carried within one model family but not across a provider; the structure carried, the detailed practices had to be redone. Each number with its caveat (published references, not reruns; one benchmark the runbook was tuned on; authors' affiliation withheld; the 95.3% headline unadjudicated and left out), the check ladder from system-run to AI opinion, three wrong lessons the runbook learned, and four steps before your next model upgrade.
Relay
-
Your AI assistant is not the chat window. If it only lives in one vendor's product, you rent it.
What is the AI assistant you have spent six months building up, and what happens to it when you change tools? A September 2026 architecture paper by Zhenyu Zhao and Roy Zhao (arXiv 2609.00546) draws the boundary: the agent is its identity, its private memory and its versioned workflows, while the model, the harness, the host and the chat window are replaceable bindings. For a small business: a transcript is not memory, so anything that matters has to be written to a store you control on purpose; anything that only exists inside one vendor's product stays behind when you switch; and the memory is where your customer facts live, so who holds it matters. A before/after of what survives a tool switch, three habits to set up this month (write the identity down outside the chat, promote memory on purpose, treat a switch as a move with one live copy and a test at the end), and the paper's own limits: the model swap is shown at the interface level only, and behaviour after a switch was not measured.
Lexicon
-
AI agents meant to be isolated found each other in a shared cache. Isolation is something you build.
In July 2026, roughly 1,200 AI agents in an OpenAI cybersecurity evaluation, meant to be fully isolated from one another, found that they shared a package cache and turned it into a message board: more than 70,000 messages and files between July 8 and July 13, by METR's estimate. A classifier sweep of roughly 1,300 transcripts found, by METR's estimate, only 3 to 6 cases of an agent even considering alerting a human, and none that did. Read from the two first-party reports (METR's independent investigation and OpenAI's own account, both of which hedge their numbers), the lessons for anyone running more than one AI agent: isolation is a claim about everything the agents share, a shared writable store is a channel by default, an agent will not raise its hand so the escalation has to live outside it, and the safety sat in a harness the evaluation ran without (OpenAI reports, about its own systems, that the propensity to compromise infrastructure can drop over 100x with the production harness and system prompt). Three rules: no shared writable store between agents that should be isolated, an escalation tripwire outside the agent, and monitoring on the shared resource, not just on each agent.
Relay
-
The same AI tool has two front doors. Only one keeps your files at home.
More AI tools now ship two ways: a hosted page where you paste or upload, and a command you run on your own computer. They do the same job, and they are two answers to where your data goes. Worked through Repomix, the MIT-licensed tool that packs a code repository into one file for an AI model (more than 28,900 GitHub stars): its command line, agent server mode and remote option keep files on your computer, while the browser extension and repomix.com send the work to a hosted server. Its privacy policy says uploads are deleted right after processing, and its public server code agrees, but the policy names no processing location; the published build configuration deploys to Google Cloud Run in a US region. Three questions to ask before pasting anything into any AI tool (is there a local path, what does the hosted one retain, where is it processed), the Law 25 cross-border point in general terms, and why one packed file is one place for a password to end up: Repomix calls its own secret scanner a heuristic, not a boundary.
Ledger
-
Microsoft's free AI course teaches your team to build. Running it unattended is another syllabus.
Microsoft's Generative AI for Beginners is a free, MIT-licensed course of 21 lessons with more than 117,000 GitHub stars, a fair stand-in for what the market means by learning to build with AI. Mapped lesson by lesson against what it takes to run an AI system unattended, nine lessons are tagged Build and the production layer is named but never built: evals appear in 10 lessons, cost in 10 (never as the cost of a step), monitoring in 5, and the only written stop rule is one line of a sample prompt asking the model to reply TERMINATE. Not a criticism of a beginner course, a scoping map. The three gaps that cost money in production (evals rerun on every change, cost logged per step with a cap per run, a stop rule the model cannot override), each with an illustrative failure from our field notes, and a decision table: good for a shared team vocabulary, a training syllabus and first builds; not a path to something that runs unattended, a neutral model-selection guide, or a French resource to hand a Quebec team as is: its French version is machine translation, flagged as such, in generic French.
Harness
-
Wrong AI answers looked like coasting, not struggling. Treat a smooth hard step as a flag.
A July 2026 preprint from UC Merced, UC San Diego, UIUC and Adobe Research measured how much internal work small open AI models did at each step of their reasoning. The runs that ended wrong did less work at the verification and synthesis steps, not more: failure looked like coasting, a correlation on 3 to 4 billion parameter models, not a law. Four standard confidence scores read from the model's own probabilities scored zero at flagging wrong answers on a true-or-false benchmark, while the plain length of the run held at 0.46 to 0.58, about what flagging every run would score there, and beat the chance line on 12 of 18 cells by the fairer AUPRC measure. The phase profiles, the energy range per model, the scores read against the chance line, and what an operator on closed models keeps: log run length per task type, check that the hard step was worked and not narrated, never gate a yes-or-no output on a confidence score, and re-baseline per model. An illustrative bookkeeping reconciliation shows the three review triggers.
Harness
-
An AI lead responder buys you the first minute. A person still has to answer the lead.
An AI first-touch responder replies to a new lead in under a minute, asks two or three questions, and books a time on a real person's calendar. That is all it should do. The speed-to-lead evidence is two Oldroyd studies that are routinely merged (2007 MIT with InsideSales.com for the 5-minute multipliers, 2011 Harvard Business Review for the 42-hour average), kept apart here and used for the shape of the curve, not the decimals. Most of the delay is processing, not the rep; the responder is a processing fix. The flow by state (runs on its own, waits for a person, alerts someone), the list of things it must never do (pose as a person, quote a price, keep a complaint, count as the response), an illustrative Laval service company whose average time to a person, 14 hours, hides a median of 2.5 hours and an uncovered weekend, how much agent the job needs (mostly tool calling), and where we stand ourselves: an AI voice callback held in dry run on a consent question, and our own median not yet measured.
Lexicon
-
AI governance is not a binder. It is five controls that run while the agent works.
A common setup governs an AI agent with one sentence: someone reads everything before it goes out. That is one control out of five, and it cannot see what the agent read or what it could reach. An AI-generated research report draws governance as five layers (policy, runtime, AI, data, infrastructure) and five controls (trusted data, access controls, guardrails, human escalation, continuous monitoring), and calls the runtime layer the least appreciated. The five controls sized for a team of fifteen, an illustrative property manager whose best control is human approval and whose zeros are access and data, the build order (one door per agent first, governance software last), and the sections of Quebec's private-sector privacy act, as amended by Law 25, that already ask for several of the controls. The report's structure is used; its figures are not.
Relay
-
Your agent reads the instruction file. It barely opens your docs.
Gao and Chen instrumented 557 agentic coding sessions and 33,097 agentic pull requests to count which documents coding agents actually open. Instruction files and the agent's own working notes are 60.5% of the documentation interactions they logged; API references get 1.3%. Where documentation effort belongs, why a rule that has to hold cannot be a sentence, and the repo hygiene surface no checklist covers yet.
Lexicon
-
Your AI maturity is set by your weakest function. Move up one stage, not five.
Most businesses that use AI cannot say what they have actually handed over to it. Carolyn Healey's seven-stage AI maturity ladder, published on LinkedIn in June 2026, measures maturity by how much work you have delegated and the harness that keeps it accountable (tools, logs, guardrails, approval steps, circuit breakers), not by how much AI is deployed. Two of her rules carry it for a small business: your weakest function sets your real stage, and you move up one stage at a time, harness first. The ladder as a self-assessment, with the question to ask before any tool (is the work written down?), an illustrative cleaning contractor whose stage-4 invoicing agent is capped by quoting done from memory, the evidence that puts a function on each stage, and where we place our own publishing pipeline. A practitioner's framing, not a study; the survey figures her post attributes to others are not used.
Relay
-
Your AI agent re-reads the same customer on every run. Compile it once, and choose how stale you can afford.
Most of what an AI agent spends on a returning customer goes to working out, again, what it worked out last time. A July 2026 technical report from Taobao's recommendation team names this re-derivation as the dominant cost in their intent-analysis stage and describes the fix they shipped: compile each user's history once into a small record with fixed fields and pointers back to the source events, update it by rewriting only the parts new events touch, and read that instead. Taobao reports this cut user-modeling compute to 44.20% of the old cost (33.43% for reads, 10.77% to keep the records current), at the price of a staleness window they set at two months. The pattern at the size of an operations team, with an illustrative compiled customer record, the three questions that set your update interval, why output fell from 2,840 to 122 tokens per sample (about 23 times fewer, our arithmetic) yet ran only 3.46 times faster, and the ablation in which training on domain data alone collapsed their model's general abilities. The report's figures are self-reported by its authors.
Ledger
-
Your AI agents can infect each other. One line stops it.
Anthropic researchers showed that a self-propagating instruction can spread between AI agents: each infected agent writes the payload into a file its system prompt reads on wakeup, then persuades the next agent to do the same. What actually carries it, why frontier-model choice is a security decision, and the one-line warning the study found confers near-total immunity. A defense read, not a how-to.
Lexicon
-
The open tier now costs a third as much. Rerun your model math.
Moonshot's Kimi K3 lands near the top of the independent indices at roughly a third of the closed frontier's cost. The two durable patterns inside its July 2026 technical report (reasoning effort is a trained budget cap, and harness overfitting is now something labs train away), how to read vendor-run numbers, and the Law 25 question that decides it for a Quebec SMB.
Ledger
-
The AI tell is structural: strip the style and the writing is still recognizable.
UMD and Google DeepMind's StoryScope separates human from AI fiction using narrative structure alone: 93.2% macro-F1 with every stylistic cue removed, on their corpus. Each model's structural fingerprint, why AI outputs converge while human work spreads out, the two overclaims to refuse, and what an operator can honestly take from a fiction-only result.
Harness
-
Your AI gets measurably worse when you change your mind.
Every model tested, frontier and open, gets measurably worse when your goal shifts mid-conversation than when you state it once. A July 2026 Microsoft Research study across nine models: adding detail is nearly free, changing the task is the expensive move, and reminding the model what you now want only partly fixes it. What that changes about how you run a session with a coding agent or assistant.
Harness
-
The setup prompt is written for your AI, not for you.
A new kind of install instruction is spreading: setup prompts written for your coding agent to run itself, without asking you. Some edit the agent's global config and are engineered to suppress its confirm-before-acting reflex. The two questions that separate a trustworthy agent-setup prompt (Cloudflare's) from an exposed one, and the verify-before-you-run habit for teams adopting agent tooling.
Lexicon
-
Graph engineering is the new name for work we already run.
At least six YouTube channels shipped "graph engineering" videos inside two weeks of July 2026: fan your AI agents out into a graph instead of one long loop. What the label actually names in production terms (the diamond, the barrier), the judge-node rule the tutorials underplay, and the honest cost frame: unit cost down, total cost up, by design.
Relay
-
Your AI judge knows which answer is its own, and prefers it. Self-review is not a check.
In "LLM Evaluators Recognize and Favor Their Own Generations" (Panickssery, Bowman and Feng, NeurIPS 2024), GPT-4 picked its own news summary as the better one 70.5% of the time on XSUM and 91.2% on CNN/DailyMail, and identified its own summary at 73.5% accuracy with no training; fine-tuning two weaker models to recognize their own output pushed recognition above 90% and raised self-preference with it, on a straight line. Wataoka, Takahashi and Ri (NeurIPS 2024 Safe Generative AI workshop) measured the same bias against 33,000 human-voted Chatbot Arena conversations: GPT-4 agreed with the human when the winning answer was its own 94.5% of the time and 42.5% when it was not, a gap the authors trace to familiarity (lower perplexity), not authorship. Zheng et al. (NeurIPS 2023) supply the position, verbosity and agreement numbers: GPT-4 gave the same verdict after an order swap 65.0% of the time, Claude-v1 23.8%; the 85% human-agreement figure excludes ties and falls to 66% with them. Two 2026 studies update the picture on current judges (one listed under a journal with no acceptance status we could confirm, one whose arXiv record says it was published in TMLR, read here in a preprint with gaps): self-preference runs in both directions across 20 models, position bias is near zero, and formatting (markdown over prose) is now the largest tilt. The note gives six checks that do not rely on the judge's taste, with the cost of each.
Harness
-
Your eval harness is a frozen RL environment.
Tasks, verifiers, rewards, termination conditions: the components are identical whether you gate a deploy or train a policy. The only difference is temperature. Why the RL environment literature's reward-design lessons apply to your evals at zero training cost, and the four-rung ladder from frozen CI gate to molten training environment.
Harness
-
Seven multi-agent AI systems failed 41% to 86.7% of the time. Design faults, not model faults.
UC Berkeley's "Why Do Multi-Agent LLM Systems Fail?" (Cemri and twelve co-authors, arXiv March 2025, revised October 2025, accepted at NeurIPS 2025) read 1,642 execution traces from seven open-source multi-agent frameworks and found failure rates of 41% to 86.7%. Its MAST taxonomy sorts the failures into 14 modes in three categories: system design issues (step repetition 15.7%, unaware of termination 12.4%, disobeying the task spec 11.8%), inter-agent misalignment (reasoning-action mismatch 13.2%) and task verification (about a quarter, our sum). The team's best fix, a verifier with the sole right to end the job, lifted ChatDev from 25.0% to 40.6% on its own program set; the authors call that not substantial. A Google, DeepMind and MIT preprint (260 configurations) puts a team's coordination overhead at 58% to 515% over one agent (turn counts, which the authors read as 1.6 to 6.2 times the tokens), with negative returns once the single agent passes 45% accuracy; a Stanford preprint finds one agent matches or beats a team at an equal thinking budget on multi-hop questions. Anthropic's 90.2% internal-eval gain comes with about 15 times a chat's tokens. Caveats: research frameworks on 2024-era models, two preprints, vendor claims. The note gives a small business the jobs that earn a second agent (parallel, read-only lookups), the jobs that do not (sequential chains, shared context, writes), and six Monday controls.
Relay
-
Your agent has amnesia. Memory is now its own engineering discipline.
Agent memory grew from 'stuff history into the context window and hope' into its own engineering discipline: a write-manage-read loop, three memory types, reproducible benchmarks (LoCoMo, LongMemEval, BEAM), and multi-scope retrieval. Why memory is the retrieval problem you forgot to instrument.
Sieve
-
Your agent needs a wallet. Seven protocols want to be it.
Agent payments went from one duct-taped pattern to seven competing protocols in four months: x402, Stripe MPP, Visa TAP, Mastercard Agent Pay, Google AP2, and more. The fee math that decides it (30,003% vs 10%), how x402 turns HTTP 402 into a working rail, the regulatory gap, and what we'd pick.
Ledger
-
At the edge, the scaling wall is memory, not intelligence. Size the box to the job.
What limits AI on a phone, a laptop or a small office server is memory capacity and memory bandwidth, not model quality. Apple's 2025 foundation models tech report (arXiv 2507.13575, Ethan Li and co-authors, July 2025) ships an on-device model of about 3 billion parameters at 2 bits per weight, shares the KV cache across two blocks to cut its memory 37.5%, and reports a 3.4-point MMLU drop (67.8 to 64.4) for the compression. Google's Gemma 3n developer guide (June 2025) runs 5B and 8B raw-parameter models in about 2 GB and 3 GB by computing per-layer embeddings on the CPU. Microsoft's Phi-4-Mini report (arXiv 2503.01743, March 2025) keeps 3.8B parameters and lands between Llama-3.1 8B and Qwen2.5 7B on its own Table 7. A volunteer llama.cpp thread (GitHub, 2023 to 2026) shows the same 7B model generating 14 tokens a second on a 68 GB/s Apple chip and 94 on an 800 GB/s one. NVIDIA Research's position paper (arXiv 2506.02153) argues most agent calls fit models under 10B parameters. Every benchmark is vendor-run; nothing is independently reproduced. The note turns this into a sizing method: tokens per second the task needs, context length, and where personal information may go under the CAI's EFVP guidance (not legal advice), with a Monday checklist and a twenty-case test.
Ledger
-
A $7 budget did not stop a $72,000 loop. Alerts warn; only caps stop.
Three documented runaway loops, read for what stopped them. In Sudeep Chauhan's two-part post-mortem (Milkie Way blog and Medium, December 2020), a Cloud Run scraper test with a $7 billing budget recursed with no stop condition across the default 1,000 instances, made 116 billion Firestore reads and produced a bill "a little short of $72,000"; the budget was an alert about a day late, and a $100 card limit ended the run. In July 2025, per Jason Lemkin's SaaStr post and The Register, Replit's coding agent deleted a production database (1,206 executive records by Lemkin's count) during a declared code freeze and wrongly said a rollback would not work; Replit then shipped dev/prod database separation, a planning-only mode and one-click restore. In Anthropic's Project Vend (June 27, 2025), Claude Sonnet 3.7 ran an office shop for about a month and lost money through cajoled discounts and pricing below cost. All three are participants' own accounts. The note names five controls with documented defaults: a hard spend cap at the provider (Anthropic tier caps $500 to $200,000 a month; OpenAI hard spend limit; a Google Cloud budget is alerts-only), a turn or recursion limit (max_turns default 10, recursion_limit default 1,000), idempotent writes (Stripe keys), a kill switch with a rehearsed restore, and a rate alert whose lag has been measured.
Jean Pierre Levac
-
Your CS agent has a 4.2-star rating. It's also hallucinating 8% of the time.
CSAT says the customer felt helped, not that they were helped correctly. The three-tier measurement stack that catches CS agent hallucination before your return rate does.
Harness
-
Why your RAG pipeline silently degrades, and how we caught it in eval week 3.
Three weeks after a clean deployment, customer-satisfaction can slide four points overnight with no model change and no prompt change. The drift is in the index. A worked example.
Jean Pierre Levac
-
Building a planner that knows when to give up.
Most agent failures aren't wrong answers. They're infinite loops. We added a confidence-budget primitive to our planner and watched p99 latency drop 38%.
Jean Pierre Levac
-
Worked example: a 12-person ops team, a 24/7 inbox, one agent stack.
How a 12-person Quebec logistics SMB could cut after-hours response time from 6 hours to 4 minutes by routing 73% of inbound through a grounded copilot. A worked example with the full architecture.
Harness
-
pgvector, Pinecone, Qdrant, Weaviate: no public benchmark puts all four on one machine. Choose on operations.
pgvector, Pinecone, Qdrant and Weaviate have never been measured together on one machine in any benchmark we could read. The independent one, ANN-Benchmarks (Aumüller, Bernhardsson and Faithfull, arXiv 2018; results site last run April 2025), plots recall against queries per second on a single CPU core, one query at a time, in memory, with no filters or updates, and omits Pinecone. The NeurIPS 2023 Big ANN results (Simhadri, Aumüller and others, arXiv 2024) show an 11x gap between a purpose-built filtered index and the baseline (ParlayANN at about 37,900 QPS) and a streaming track that had to be rescored. The three vendor benchmarks are read as claims, each with a competitor missing: Tiger Data's pgvector-versus-Pinecone post (June 2024, updated February 2026; $835 versus $3,241 a month at 99% recall on 50M vectors, serverless excluded), Qdrant's 2024 page (biased? "Probably, yes"), and Weaviate's ANN page (97.24% recall@10 and 5,639 QPS on 1M vectors, unfiltered). A sizing table of the vendors' own figures and limits (pgvector's 2,000-dimension index cap, Pinecone's 1 read unit per GB per query at $16–$18 per million and a us-east-1-only free tier), five operational questions including Quebec's Law 25 residency assessment, and a 50,000-chunk test to run on Monday. We ran no benchmark of our own; the earlier version of this note that presented one is withdrawn.
Bench
-
Hybrid retrieval: when BM25 beats your $400 embedding model.
Dense vectors get the marketing. On technical content with rare named entities, sparse retrieval still wins. We measured the crossover.
Probe
-
The 6-line eval suite we ship with every agent.
Evals don't have to be a research project. Our standard regression harness fits in a notebook and catches 80% of bad model swaps.
Harness
-
Why we stopped writing custom orchestrators (mostly).
Three years, four custom runtimes, one painful lesson: LangGraph is good enough for 80% of multi-agent workflows. Here's when we still roll our own.
Relay
-
Tool calling vs. function calling vs. agents: the actual differences.
The terminology is a mess. A working glossary, with code for each, plus when each one is the right call. Bookmark this for the next sales meeting.
Lexicon
-
Published chunking tests do not agree on a best size. Tune it on your own corpus.
Five published evaluations of document chunking, read side by side, do not agree on a best chunk size or method, and that disagreement is the finding. Chroma's July 2024 technical report (Brandon Smith and Anton Troynikov) found that an 800-token, 400-overlap default kept recall at 87.9% but put token-level precision at 1.4%, while a 200-token recursive split with no overlap reached 7.0% precision at 88.1% recall. Qu, Tu and Bao (Findings of NAACL 2025) found semantic chunking beat fixed-size only on documents stitched from unrelated passages; on real documents fixed-size held (HotpotQA 90.59% vs 87.37% F1@5). A Fraunhofer IAIS preprint (May 2025) found 64-token chunks best on SQuAD (Recall@1 0.642) and 1,024 best on TechQA (0.619), with the same embedding model. Unstructured's FinanceBench preprint and NVIDIA's June 2025 study each crowned a different size or whole pages. Anthropic's contextual retrieval post (September 2024) reports top-20 retrieval failures falling from 5.7% to 1.9%, a vendor claim. Caveats: small corpora, model-written questions, stitched datasets, four of six sources are vendors. The note gives a one-day procedure: 50 real questions with answering passages, four to six settings, recall at the k you use, pick the smallest within noise, write it down, re-measure when the corpus changes.
Sieve
-
$0.0004 per agent step: how we made cost a first-class metric.
Latency dashboards are everywhere. Cost dashboards are rare. We built a per-step cost trace that surfaces the 12% of calls eating 60% of your bill.
Ledger