Post-mortem: the loop that cost $3,200 overnight.

Consider a research agent that enters an undetected retry loop after receiving an ambiguous tool response. With no termination condition beyond a per-session step cap (which the loop resets on each new sub-task) and no cost alerting in place, the agent makes 24,847 API calls over 9 hours and 14 minutes, accumulating $3,218 in API spend before an engineer notices the anomaly during a routine morning check. This post walks that failure end to end.

Minute by minute, then hour by hour

The incident timeline below records every state change from normal invocation to containment. Spend figures are the running API cost at each checkpoint.

The spend curve, hour by hour

Total API spend over the incident window. The loop warms up after 23:00, plateaus near $400/hour through the night, and is cut off by manual termination at 07:18.

Three missing safeguards. All three required.

This was not a rare edge case. The trigger, an empty 200 response from a rate-limited scraper, happens on roughly 3% of scraper invocations. The loop was a foreseeable outcome of a foreseeable input, with no defense in depth. The post-mortem identified five contributing factors. Three were independent safeguards that, had any one of them been in place, would have contained the incident.

What the code looked like. Before and after.

The missing termination condition was three lines: a workflow-level step counter and a check before each sub-task spawn. The cost alert was a single CloudWatch rule. Neither required architectural changes. Both required the incident to make them feel necessary.

# ── BEFORE: sub-task spawn with no workflow-level guard ──────────
def spawn_subtask(task: str, context: dict) -> dict:
    # No check on total workflow steps. This is the gap.
    sub_agent = ResearchAgent(max_steps=20)  # per-task cap only
    return sub_agent.run(task, context)

# ── AFTER: workflow-level budget enforced across all sub-tasks ────
@dataclass
class WorkflowBudget:
    max_steps:     int   = 100   # across ALL sub-tasks
    max_spend_usd: float = 5.00  # hard dollar cap
    steps_used:    int   = 0
    spend_usd:     float = 0.0

    def check(self) -> None:
        if self.steps_used >= self.max_steps:
            raise WorkflowBudgetExceeded(f"Step limit  reached")
        if self.spend_usd >= self.max_spend_usd:
            raise WorkflowBudgetExceeded(f"Spend limit $ reached")

def spawn_subtask(task: str, context: dict, budget: WorkflowBudget) -> dict:
    budget.check()                           # enforced before spawn
    sub_agent = ResearchAgent(
        max_steps=20,
        budget=budget,                        # shared budget object
        on_step=lambda cost: setattr(
            budget, "spend_usd", budget.spend_usd + cost
        )
    )
    return sub_agent.run(task, context)

# The confidence budget (the planner post) sits above this. It
# detects stalling before the dollar cap is reached. The dollar
# cap is the last line of defense, not the first.

The WorkflowBudget is passed down through every sub-task spawn. Sub-tasks don't get a fresh budget, they draw from the same pool. The dollar cap is set conservatively low for the first deploy of any new workflow and raised as confidence in the workflow's expected spend range grows. The confidence budget from the planner post sits upstream and typically catches runaway loops before they hit the dollar cap; the dollar cap is the last-resort termination.

What changed. All of it.

Three things this incident changed permanently

Defense in depth is not optional for agentic loops. Any single safeguard can be routed around: the step cap was. The confidence budget, the workflow-level spend cap, and the real-time cost alert are three independent defenses. If one is missing, the others can still contain an incident. If all three are missing, you're writing a post-mortem.

Invisible success is the hardest failure mode to detect. The loop made 24,847 successful API calls. Every call returned 200. No error was thrown. The monitoring stack saw a healthy system. The only anomalous signal was spend, which nobody was watching. Success metrics don't catch loops; they confirm them.

The safeguards in this series are listed in order of incident prevention, not implementation complexity. The confidence budget was written before this incident happened but hadn't been deployed to all workflows. The eval harness existed but didn't cover the failure mode. The cost tracer didn't exist yet. All three posts were written, at least in part, because of this night.

If you would like us to pressure-test the safeguards on your own agent stack, the contact form is the fastest way. We do 30-minute reviews for production agent stacks, free.