How do you stop an agent from looping or running away?

Budgets belong in the runtime, not the prompt

Telling a model to be efficient is a preference, not a limit. Containment has to live in the loop that calls it, and there are four independent budgets because they fail independently: a step count, a cumulative token count across the whole run, wall-clock time both per tool call and per run, and a monetary ceiling. A run can sit well inside its step budget and still spend a fortune, because one step retrieved forty thousand tokens of context that every subsequent step now re-sends. Counting steps alone is the most common form of false comfort.

Set them per run, and also per tenant per day, because the failure you will actually get paged for is not one pathological run but a scheduled job that starts three thousand of them. The daily ceiling is the one that turns an incident into a throttle.

What happens at exhaustion matters as much as the limit. An agent that hits a budget and simply stops has wasted everything it did. The right behaviour is to halt the loop, summarise what was accomplished, name what remains, and return that as a result with a reason code — which also gives you a metric worth watching, because a rising share of budget-exhausted runs is a product signal, not just an operational one.

Repetition and lack of progress are different faults

The obvious loop is the agent calling the same tool with the same arguments repeatedly, usually because the tool keeps returning an error it cannot act on. Detect it cheaply: normalise the arguments, hash them with the tool name, and stop or intervene after a small number of identical calls. Two-cycles are the near variant, where the agent alternates between two tools each of which undoes the other's premise, so track a short window rather than only the immediately preceding call.

The subtler fault is a run that never repeats itself and never gets anywhere: each step is novel, each result is unhelpful, and the agent keeps generating plausible next actions. Repetition detection is blind to this. What catches it is a progress signal — hash the tool results and count how many recent steps produced information the run had not already seen, or maintain an explicit measure of whether the stated goal is closer. When a window of steps yields no new information, escalate rather than continue, because more steps of the same character will not help.

The rule underneath both is that something must change between attempts. If a tool returned an error and the next call is identical, the model has not incorporated the error and further attempts are wasted; require a changed argument, a different tool, or a stop.

flowchart TD
  A[Step requested] --> B{Budgets remaining}
  B -->|no| H[Halt with partial result]
  B -->|yes| C{Repeat or no progress}
  C -->|yes| H
  C -->|no| D{Irreversible action}
  D -->|yes| E[Await approval bound to arguments]
  D -->|no| F[Execute tool]
  E --> F
  F --> A

Note that the approval gate sits inside the loop rather than at the start of the run. A plan approved up front is not a control, because the plan is what changes.

Timeouts at every level, and cancellation that reaches the tool

Three timeouts, not one. Each tool call gets its own, tight enough that a hanging dependency does not consume the run's wall-clock allowance. Each model call gets one. The run gets an overall deadline, checked between steps, after which it halts regardless of what the individual calls are doing.

Cancellation is the part that is usually missing. A run needs an identifier, a cancellable state that the loop checks between steps, and tools that propagate cancellation to whatever they called. Between-step checking is enough for most purposes and is honest about its granularity: a call already in flight against a third party will complete, which is precisely why mutating tools need idempotency keys and why the recovery path after cancellation is a read-back of state rather than an assumption that nothing happened. You also want a coarser control — the ability to disable a tool or a whole class of runs by configuration — because the fastest mitigation during an incident is removing the capability, not chasing the runs.

Sub-agents inherit the budget, they do not get a fresh one

Any design where an agent can spawn another has to answer this explicitly, and the naive implementation is where cost accidents come from. If each child receives the default budget, total spend is the branching factor to the power of the depth, and the parent's limit means nothing. Budgets must be held by the run tree and drawn down by every node in it, depth must be capped, and concurrent fan-out must be capped separately, because ten parallel children can exhaust a shared budget before any of the checks between the parent's steps run. Cancellation propagates down the same tree.

An approval gate that cannot be recycled

Classify tools by reversibility and blast radius. Reading is free. Writes to your own systems with a straightforward undo are cheap. Sending communications, moving money, deleting data, deploying, or anything touching a third party is irreversible in the sense that matters: no compensating action fully restores the previous state.

For that last category the run pauses and a human confirms. Two properties make the gate real. The approval is presented with the concrete, resolved arguments — the exact recipients, the exact amount, the exact rows — because approving an intention is not approving an action. And the approval is bound to those arguments, typically by hashing them, so it authorises that call and not the next one; otherwise an agent that revises its plan after approval proceeds with a licence granted for something else. Blanket approval for a session is the same defect with better ergonomics.

Then put a second, independent limit at the tool itself: a per-run and per-day cap on refunds issued or messages sent, enforced by the service rather than the orchestrator. The orchestrator is code you will get wrong at some point, and a limit that only exists on the caller's side is not a limit.

Make the run legible while it is running

You cannot control what you cannot see. Emit a structured trace per run — each step's tool, arguments, result size, latency, token and cost draw, and the running totals — and expose it in something an engineer can read during an incident rather than reconstruct afterwards from log lines. Alert on distribution shifts rather than absolutes: the share of runs terminating on budget, the ninety-fifth percentile step count, the rate of repeated-call interventions, and spend per run against its usual band. Every runaway has a warning period during which those numbers move, and the whole point of the budgets is that the bill they cap is one you notice before finance does.