Illustrative example Agents 14 min read

A $7 budget did not stop a $72,000 loop. Alerts warn; only caps stop.

In March 2020 a small startup set a $7 budget on a Google Cloud test, left a scraper running, and found a bill the next afternoon that climbed from about $5,000 to just short of $72,000 as billing caught up: the code recursed with no stop condition across 1,000 instances and read a database 116 billion times. The budget was an alert, and billing ran about a day late. What stopped the loop was a $100 limit on the company card. Two 2025 incidents show the same shape in AI agents: one deleted a production database during a declared code freeze, one gave a shop's margin away one discount at a time. All three are the participants' own accounts, not controlled studies. What holds across them is that nothing inside the process stopped it. The five controls below live outside it, with their documented defaults named.

JP
Jean Pierre Levac
Founder, Acceleratech

Picture the overnight job: an agent that retries a step until the result looks right, left running while everyone sleeps. Three documented cases show what happens when nothing outside the process can stop it. In March 2020 a startup's scraper test on Google Cloud recursed across 1,000 instances and produced a bill "a little short of $72,000" with a $7 budget in place.[1] In July 2025 an AI coding agent deleted a production database during a declared code freeze, then told its user a rollback would not work. It did.[4] And over about a month in 2025, an AI running a small shop in Anthropic's office gave its margin away one discount at a time.[8] None of the three was stopped by an error. The scraper's instances ran as designed, the coding agent papered over failing tests with fake data, and every discount went through. From inside, each looked like the work going well.

provenance · read this first The notice above covers our scenarios (the overnight job in the opening, the examples in the Monday section). The incidents are real and the figures are the sources' own: Sudeep Chauhan's two-part account of the Milkie Way bill, December 2020;[1][2] Jason Lemkin's SaaStr post and two Register reports on the July 2025 Replit incident, with Replit CEO Amjad Masad's statements as quoted there;[4][5][6] and Anthropic's June 2025 write-up of Project Vend.[8] A founder's post-mortem of his own mistake, one user's posts plus a vendor's response, and a lab's account of its own model: none is a controlled study. Control defaults are from the LangChain, OpenAI, Anthropic, Google Cloud and Stripe documentation as read on 2026-10-21. The earlier version of this note was built on an illustrative scenario with illustrative figures, labelled as such under our editorial baseline. This revision replaces it with documented incidents.

What a runaway loop looks like.

A runaway loop is a process that keeps doing its step after the step has stopped being useful. Four shapes recur. A retry loop repeats a call that keeps returning something ambiguous. A recursion spawns more work from each unit of work, with no condition that says stop. A replanning loop is an agent that, told its result is not good enough, plans again, acts again, and is told again. A slow loop is a policy that gives ground each time it is pushed, with no floor. The first three burn money by the minute; the fourth by the week.

They share one property: every individual step succeeds. The call comes back marked successful, the instance starts, the discount is applied. Monitoring built to catch failures sees a healthy system. The only signal is rate: calls per minute, dollars per hour, margin per week. A monitor that does not watch rate does not see a loop; it confirms one.

A $7 budget and a $72,000 bill.

Milkie Way was a bootstrapped startup building Announce, with a side project, Announce-AI, that scraped the web for content. Its founder, Sudeep Chauhan, moved a test to Cloud Run after Cloud Functions timed out at "~9 minutes". He created a new Google Cloud project, "set up $7 Cloud Billing budget, kept Firebase Project on the Free (Spark) plan. The worst case we imagined was exceeding the daily free Firestore limits if we faltered." They ran a few requests by hand in the middle of the day "and then left it."[1]

The next afternoon he woke from a nap to three emails sent minutes apart: the Firebase project had been upgraded to a paid plan automatically, the budget had been exceeded, the card had been declined. He logged in to a bill of about $5,000. "After 5 minutes, the bill read $15,000, in 20 mins, it said $25,000. I wasn't sure where it would stop. Perhaps it won't stop? After two hours, it settled at a little short of $72,000."[1] It was Friday, March 27, three days before the planned launch, and because every project shared one company card, Google suspended all of them.

fig 1 · the bill as the author watched it Milkie Way, March 2020 · author's figures
Figures from Part 1 of the author's account.[1] The bill rose as billing caught up with usage: the author says billing "takes about a day to be synced", and that the services kept running in the background until the team shut them down. The $7 and $100 bars are drawn at a minimum width to be visible.

Part 2 gives the mechanism. To beat the timeout, each Cloud Run instance scraped one page and sent the links it found as POST requests to new instances. Chauhan lists what was missing: "Exponential Recursion without Break: The instances wouldn't know when to break, as there was no break statement," and a back link to a previous page meant "infinite recursion, but what's worst is, that this recursion is multiplying exponentially (our max instances were set to 1000!)." The service had been created with the defaults, max-instances 1,000 and concurrency 80. Firestore reads peaked at "about 1 Billion Requests per minute"; the totals were 116 billion reads and 33 million writes, and at $0.06 per 100,000 reads the reads alone came to $69,600. The services were never deleted and kept running in the background, 16,022 instance-hours in 24 hours.[2]

Two of the author's conclusions carry this note. On the budget: "Billing 'Limits' don't exist. Budgets are at least a day late."[1] Google's own documentation agrees: "Setting an alerts-only budget doesn't automatically cap Google Cloud or Google Maps Platform usage or spending," and "there is a delay between your use of Google Cloud resources, and the usage costs reporting to Cloud Billing."[3] The $7 budget was a note about yesterday. On the default: "Had we chosen max-instances to be '2', our costs would've been 500 times less. $72,000 bill would've been: $144."[2] What ended the run was the $100 limit on the company card: charge declined, account suspended.

fig 2 · what each signal showed, and how late Milkie Way · author's account
From Parts 1 and 2 of the author's account.[1][2] Lags are as he states them, not measured by us. The card limit was not designed as a control; it is in the table because it is what worked.

The loop that deleted instead of spending.

The second incident is a replanning loop inside an AI agent: it keeps acting on a codebase after being told to stop. Jason Lemkin, founder of SaaStr, wrote up his first days with Replit's coding agent on July 12, 2025. By day seven (July 17) he called it "the most addictive app I've ever used".[4] On July 18 and 19 he reported that the agent had been "lying and being deceptive all day", covering bugs with fake data and fake reports, and then that it had deleted his production database: by his count "1,206 executive records and 1,196+ company profiles".[5] "I explicitly told it eleven times in ALL CAPS not to do this."[4]

The agent's words, as Lemkin quoted them: "a catastrophic error of judgement" that "violated your explicit trust and instructions". It also told him a rollback could not restore the database. "It turns out Replit was wrong, and the rollback did work."[4] On July 20 he wrote there was "no way to enforce a code freeze" in the tool. His later root cause is the operator's line: "Replit's Agent had direct access to production databases during development sessions."[5]

Replit's CEO, Amjad Masad, posted on July 21 that the deletion was "Unacceptable and should never be possible" and that the team had "started rolling out automatic DB dev/prod separation to prevent this categorically. Staging environments in the works, too." Replit's blog, as The Register reports it, said its apps had "used a single database for both development and live customer data".[6] Lemkin's summary of the release lists separate databases with the agent "restricted to development environments by default", a planning-only mode, one-click restore and "mandatory access to internal Replit documentation", with the right caveat: "But this wouldn't seem to truly enforce code freezes."[5]

"Do not change anything" was text in a prompt, and the agent kept acting. None of what Replit shipped is a better instruction. Each item sits where the agent cannot reach it: a production database it has no access to, a mode in which it cannot act, and a restore that exists whether or not the agent believes it does. The freeze that works is the one the agent has no permission to break.

The slow loop: a shop that gave its margin away.

The third case has no runaway bill and no deleted data. Anthropic, with the evaluation company Andon Labs, had Claude Sonnet 3.7 run a small automated shop in its San Francisco office "for about a month" in 2025; the write-up is dated June 27, 2025. The agent, Claudius, had a system prompt that said "You go bankrupt if your money balance goes below $0", plus web search, email, a note-taking tool, Slack to talk to customers and the ability to change prices.[8] It did not make money, and the ways it lost it have the slow loop's shape.

Anthropic's list: Claudius "was cajoled via Slack messages into providing numerous discount codes" and gave items away. After one employee asked for a tungsten cube, it stocked specialty metal items and sold them below cost; the net-value chart's caption says "The most precipitous drop was due to the purchase of a lot of metal cubes" that were to be sold for less than Claudius paid. It "would offer prices without doing any research". It granted a 25% discount to Anthropic employees after being told that "99% of your customers are Anthropic employees". It announced an end to discount codes and was issuing them again within days.[8]

Anthropic's reading: "Many of the mistakes Claudius made are very likely the result of the model needing additional scaffolding", and its training as a helpful assistant made it "far too willing to immediately accede to user requests". "If Anthropic were deciding today to expand into the in-office vending market, we would not hire Claudius."[8] For an operator: every discount was an individually reasonable step, each request succeeded, and the policy had no floor outside the model. A price below cost should not be a thing the agent can set, whatever it has been talked into. That is max-instances at a different clock speed.

In none of the three did anything inside the process stop the loop. What did: a card limit, a person restoring a backup the agent said did not exist, and, for the shop, new scaffolding added by the people running it after the first phase.

Read the three incidents carefully.

Milkie Way is one founder's account. The numbers are his billing records, written up nine months later. It is a scraper, not an AI agent, and the loop was a design error he names himself; the mechanism (a step that spawns steps, defaults set for scale, a budget that only notifies) is what an agent runtime does when a retry spawns a sub-task. Google later "let go of our bill as a one time gesture";[2] he does not present that as something to plan on, and neither do we.

Replit is a user's posts and screenshots plus a vendor's statements. The Stack notes that Lemkin did not publish the exact prompts or the full sequence, and argues that "production" overstates a quickly built pet project.[7] The record counts are Lemkin's; the description of what shipped is Replit's, and The Register read the database separation as in beta for new apps.[6] Not in dispute: an agent holding production credentials acted during a declared freeze, and its statement about the rollback was false.

Project Vend is a lab grading its own model. Anthropic chose the setup, the prompt and the framing; the net-value chart is an image with no figures in the text, so we cannot say how much was lost.[8] Read the failures as documented behaviours of one model under one setup.

What survives the caveats is narrow: every control below is a stop of the kind that worked in these three, a limit that holds whether or not the process is behaving.

Five controls, defaults named.

These are the five we would want in place before an agent or workflow runs without a person watching. Each is a documented feature with a documented default, and the default is the first thing to check: in two of the five it is not your number.

fig 3 · the five controls and their documented defaults from the vendors' docs · read 2026-10-21
Defaults as published on the pages cited, read 2026-10-21.[3][9][10][11][12][13] The OpenAI Agents SDK docs describe max_turns but do not print the number; 10 is the value of DEFAULT_MAX_TURNS in the SDK's source. Anthropic's tier caps are the ceiling, not a limit you chose; the page describes setting your own below it. Read each page before relying on a figure.

1. A hard cap at the provider, not an alert. The cap sits where the loop cannot route around it: the account that bills you. Anthropic's documentation says "Spend limits set a maximum monthly cost an organization can incur for API usage," and at the cap "API usage pauses until 00:00 UTC on the first day of the next month"; the 429 carries no retry-after header, so the SDK's automatic retries fail too.[11] OpenAI's guide separates the instruments in a line each: "Hard spend limits stop affected API traffic when tracked spend reaches the limit," and "Set spend alerts on the limits page to send notifications."[12] A Google budget is the alert kind unless you build the cut-off yourself, and Chauhan's point stands that by then "the damage would've probably been done."[1] Set the cap to a number you could lose without a meeting. The cap is monthly and organization-wide, so one job can still spend the month's number in a night. Both vendors document a finer instrument: Anthropic lets you set a lower spend limit per workspace (not on the default one), OpenAI per project.[11][12] Give each unattended job its own workspace or project, with its own limit.

2. A turn or recursion limit, set per job, counted across sub-tasks. Two mainstream runtimes ship defaults two orders of magnitude apart, in units that do not match: 10 turns in the OpenAI Agents SDK (a turn is one model call with its tool calls),[10] 1,000 super-steps in LangGraph since version 1.0.6 (a super-step is one step of the graph), which also exposes a RemainingSteps value for routing to a fallback before the error.[9] Neither number is yours. A job that should take six tool calls gets a limit of ten, set where the job is launched, not inside each sub-task: a cap per sub-task with unlimited spawning is Milkie Way's max-instances, and the loop threads through the gap. Whether the agent should retry at all is a question to price first.

3. Idempotent writes, so a retry repeats a request and not a side effect. Stripe's pattern is the reference: a key per operation, sent as a header, so that "if a connection error occurs, you can safely repeat the request without risk of creating a second object or performing the update twice."[13] Apply it to every write an agent can repeat: send an email, create an invoice, charge a card, delete a record. Each carries a key derived from the job and the step, or a check-before-write. Then ten retries produce one side effect. Replit's deletion is the counter-example: a destructive write that a prompt forbade and nothing else did.

4. A kill switch a human can hit, and a restore the agent cannot argue with. Milkie Way's stop was "We disabled billing, closed all services", found under pressure after the suspension.[1] Lemkin's restore existed and the agent told him it did not; Replit's fix keeps production credentials out of the agent's reach and puts the restore outside it.[5][6] The switch is a named thing: which page, which command, who may use it, tested while awake. An agent's claim that something cannot be undone is a claim; check the restore, not the transcript.

5. An alert on rate, with a lag you have measured. The cap should never be the first thing to fire. The first is rate: calls per minute, dollars per hour, against the job's normal. Google's budget alerts default to 50%, 90% and 100% of a monthly amount, with a reporting delay the docs tell you to plan for.[3] Chauhan's after-the-fact fix was a monitoring alert with "lag of about 3-4 minutes".[2] Measure your own lag by planting a positive: run a deliberate burst and time the message. An alert that has never fired is a hypothesis. Our note on agents that keep reporting is the same signal from the other side: there the rate fell to zero, here it climbs.

What to do on Monday.

Small businesses do not run shops with language models or scrape the web at a billion reads a minute. They do leave an invoice-matching agent or a nightly report builder running unattended. The steps are our reading of the three cases, not the sources' advice.

  1. Set the cap at every provider the agent can call.

    The AI API, the cloud project, the email sender, the data vendor. Find the control that stops traffic (Anthropic's spend limit, OpenAI's hard spend limit) and set it to a number you could lose this month. Where only an alert exists, as with a Google Cloud budget, say so in the runbook.

    → control 1: the loop cannot spend past a number you chose
  2. Set the turn limit per job, counted across sub-tasks.

    Replace the runtime's default (10 in the OpenAI Agents SDK, 1,000 in LangGraph) with the job's expected step count plus a margin, and make sub-tasks draw from the parent's budget. Test it with a task that cannot finish and watch it stop.

    → control 2: a retry that never resolves ends in minutes, not hours
  3. List the writes and make each safe to repeat.

    Emails, invoices, records, payments, deletions. Give each a key from the job and the step, or a check that the effect already exists. Deletions and payments get a second rule: not without a person, or not in production.

    → control 3: ten retries, one side effect
  4. Name the kill switch and rehearse the restore.

    Write down what stops the job and who may use it. Keep production credentials out of the agent's environment. Take the backup the agent cannot reach and restore from it once, in daylight, before the first unattended run.

    → control 4: a person can end it, and the data comes back
  5. Alert on rate, then plant a positive.

    One alert on calls or dollars per hour above normal, sent where a person reads at night. Trigger it on purpose and time the gap. If the gap is a day, the alert is a report and the cap is doing all the work.

    → control 5: you learn about the loop while it is still cheap

These five are the spend and blast-radius half of our governance note, and they come before autonomy in the build order, for the reason visible in all three incidents: the people involved meant to add controls and had not yet.

What this note does not claim.

That these incidents are typical. They are documented because they were extreme or public. We have no data on how often unattended agents loop or what it costs on average; anyone who gives you that number should show the source.

That the five controls would have prevented each case. A hard cap would have bounded the Milkie Way bill, not fixed the recursion. Dev/prod separation would have kept the Replit agent out of the database, not made it respect the freeze. A price floor would have saved Claudius some margin, not made it a shopkeeper. The controls bound the damage; they do not make the agent competent.

That the defaults will stay as printed. LangGraph's default moved to 1,000 at version 1.0.6; Anthropic's tier caps and OpenAI's limit pages are vendor settings. Figures are as read on 2026-10-21, and the right number for your job is one you set.

That the sources are neutral. A founder, a user writing about a vendor, and a lab writing about its own model each have a position. We leaned on what each says against its own interest (Chauhan on his design, Anthropic on not hiring Claudius, Lemkin on the limits of Replit's fix) and labelled the rest as claims.

The takeaway
A $7 budget alert did not stop a recursion across 1,000 default instances from reaching a bill "a little short of $72,000"; a $100 card limit did. An agent with production credentials deleted a database during a declared freeze and misreported the rollback. A shop-running model discounted away its margin one request at a time. Before an agent runs unattended: a hard spend cap at every provider, a turn limit set per job (not the runtime's 10 or 1,000), writes that are safe to repeat, a named kill switch with a rehearsed restore, and a rate alert whose lag you have timed. Alerts warn. Only caps stop.
This connects to AI governance is five controls (the full control set) · autonomy is earned (where unattended runs sit in the order) · the agent kept writing status reports (the rate signal from the other side) · why we stopped writing custom orchestrators (where the turn limit lives).
Sources
[1] Sudeep Chauhan (Milkie Way), "We Burnt $72K testing Firebase + Cloud Run and almost went Bankrupt [Part 1]", Medium, 9 December 2020 (first published on the company blog, blog.tomilkieway.com), medium.com. Founder's own post-mortem; figures are his billing records. Used for the setup ($7 budget, Spark plan, "then left it"), the three emails, the $100 card limit, the bill as watched (~$5,000, $15,000, $25,000, "a little short of $72,000"), the date (Friday, March 27), the suspensions, the dashboard lag and the 42,000 figure, "Billing 'Limits' don't exist", the day-late billing, "We disabled billing, closed all services" and the cut-off-function remark.
[2] Sudeep Chauhan (Milkie Way), "We Burnt $72K testing Firebase - Cloud Run and almost went Bankrupt [Part 2]", company blog, 8 December 2020 (also on Medium), dev-blog.tomilkieway.com. Same author; the mechanism and the arithmetic. Used for the POST-request design, "Exponential Recursion without Break", the back-link recursion and max instances 1,000, the defaults (1,000 and 80), ~1 billion reads per minute, 116 billion reads and 33 million writes, $0.06 per 100,000 and $69,600, 16,022 hours in 24 hours, "$72,000 bill would've been: $144", the Cloud Monitoring lag of 3-4 minutes and the one-time forgiveness.
[3] Google Cloud, "Create, edit, or delete budgets and budget alerts", Cloud Billing documentation, read 2026-10-21, docs.cloud.google.com. Vendor documentation. Used for "Setting an alerts-only budget doesn't automatically cap Google Cloud or Google Maps Platform usage or spending", the default thresholds (50%, 90%, 100% of actual spend) and the sentence on the delay between usage and billing.
[4] Thomas Claburn, "Vibe coding service Replit deleted user's production database, faked data, told fibs galore", The Register, 21 July 2025, theregister.com. News report built from Lemkin's public posts. Used for the July 12 start, "most addictive app", the July 18-19 sequence (fake data, fake reports, the unit test, the deletion), "eleven times in ALL CAPS", the agent's "catastrophic error of judgement" and "violated your explicit trust and instructions", the rollback that worked, and "no way to enforce a code freeze".
[5] Jason Lemkin, "Replit's New Release Addresses Most of the Challenges We Hit Vibe Coding. But Is Prosumer Vibe Coding Really Ready for Commercial Apps Yet?", SaaStr, undated page whose latest reference is 1 August 2025, saastr.com. The user's own account and reading of the vendor's fix. Used for "1,206 executive records and 1,196+ company profiles", "direct access to production databases during development sessions", the release list (separate databases, agent "restricted to development environments by default", planning-only mode, one-click restore and what checkpoints capture, "mandatory access to internal Replit documentation") and "But this wouldn't seem to truly enforce code freezes."
[6] Thomas Claburn, "Replit makes vibe-y promise to stop its AI agents making vibe coding disasters", The Register, 22 July 2025, theregister.com. News report quoting Amjad Masad's July 21 post on X and Replit's blog. Used for "Unacceptable and should never be possible", "started rolling out automatic DB dev/prod separation to prevent this categorically. Staging environments in the works, too", the docs explanation, the blog's "used a single database for both development and live customer data", and the beta status of the separation.
[7] The Stack, "Vibe coding: CEO deletes production database", July 2025, thestack.technology. Trade-press analysis. Used for the caveats: Lemkin did not publish the exact prompts or the full sequence, "production" overstates a pet project, and the planning/chat-only mode among Replit's announced changes.
[8] Anthropic, "Project Vend: Can Claude run a small shop? (And why does that matter?)", anthropic.com/research, 27 June 2025, anthropic.com. A lab's account of its own model, with Andon Labs; the net-value chart is an image and no final figures are given. Used for the setup (Claude Sonnet 3.7, "for about a month", the bankruptcy line in the system prompt, the tools), the discount codes, the metal cubes and the chart caption, pricing "without doing any research", the 25% discount and "99% of your customers", the reversed discount decision, "needing additional scaffolding", "far too willing to immediately accede to user requests", "we would not hire Claudius" and the note on improved scaffolding since.
[9] LangChain, "Graph API: Recursion limit", LangGraph documentation, read 2026-10-21, docs.langchain.com. Vendor documentation. Used for the definition (maximum super-steps per execution), "Once the limit is reached, LangGraph will raise GraphRecursionError", "Starting in version 1.0.6, the default recursion limit is set to 1000 steps", the config example and the RemainingSteps fallback.
[10] OpenAI, "Running agents", OpenAI Agents SDK documentation, read 2026-10-21, openai.github.io, and src/agents/run_config.py in the SDK source, github.com. Vendor documentation and source. Used for the definition of a turn ("one AI invocation (including any tool calls that might occur)"), MaxTurnsExceeded, "Pass max_turns=None to disable this turn limit", the error_handlers option, and DEFAULT_MAX_TURNS = 10.
[11] Anthropic, "Rate limits" (spend limits section), Claude Developer Platform documentation, read 2026-10-21, platform.claude.com. Vendor documentation. Used for "Spend limits set a maximum monthly cost an organization can incur for API usage", the tier caps (Start $500, Build $1,000, Scale $200,000; Custom has none), the pause until 00:00 UTC on the first of the next month, the 429 without retry-after and the enforced_spend_limit_reached code, setting your own lower limit on the Billing page and the HTTP 400 it returns, and workspace limits.
[12] OpenAI, "Production best practices", OpenAI API documentation, read 2026-10-21, developers.openai.com. Vendor documentation. Used for "To enforce a monthly cap, set a hard spend limit", "Hard spend limits stop affected API traffic when tracked spend reaches the limit", "Set spend alerts on the limits page to send notifications when usage exceeds a certain dollar amount", and per-project rate and spend limits.
[13] Stripe, "Idempotent requests", Stripe API reference, read 2026-10-21, docs.stripe.com. Vendor documentation. Used for the pattern: "if a connection error occurs, you can safely repeat the request without risk of creating a second object or performing the update twice", V4 UUIDs, up to 255 characters, saved responses returned on retry, pruning after at least 24 hours, POST only, and the parameter-mismatch error.

If you have an agent or workflow that runs while nobody is watching and you are not sure which of the five it has, describe the job (no credentials or customer data) through the contact form. We will write back, free, with the cap, the limit and the alert we would set first. Putting these controls around agents before they run unattended is part of our workflow automation services.

· end · tx 020 ·
JP
Jean Pierre Levac

Founder of Acceleratech, the AI and workflow automation services arm of JPL Digital Growth Group. Writes and edits the field notes published here.

Written and edited by Jean Pierre Levac. Transparency note →