Design a platform that runs agent tasks for hours and pauses for a human to approve risky steps.
Use durable execution. Persist every model output and tool result as a step in a log, so a run resumes on any worker by replaying recorded results instead of asking the model again. An approval wait suspends the run with no thread held, and every tool is idempotent on a run-and-step key.
6 min readUpdated 2026-10-10
0
Answer
Short answer
Treat each run as durable execution, not as a process that happens to live a long time. Write every model output and every tool result to a step log before moving on, so any worker can resume the run by replaying what was recorded. A human-in-the-loop approval then becomes a stored state with an expiry, not a thread waiting on a person. Make every tool idempotent on a key derived from the run and step.
Was this answer helpful?
Likely follow-ups
→You deploy a new prompt while two hundred runs are suspended. Which version does each one resume on?
→The approver wants to edit the amount rather than approve or reject. How does that enter the run?
→A run has been going for six hours and its transcript no longer fits the context window. What do you carry forward?
→How would you test that a run survives a worker being killed at every possible step?
An agent task that runs for hours will outlive the process that started it. Workers are redeployed, autoscaled away, and killed by the platform. A human-in-the-loop approval might take an afternoon. If the run's state is the call stack of one worker, any of those events loses it.
The waiting also costs capacity. Assume 600 runs a day pause for approval and the average wait is four hours. Little's law says the number of runs waiting at any moment is arrival rate times average wait: 600 / 24 × 4 = 100. If each waiting run holds a worker thread or a container, you are paying for a hundred workers to do nothing. Under durable execution those hundred runs are rows in a table, and no worker holds them.
The step log is the run
Model the run as an append-only sequence of steps. Each step records the model's response, the tool call it proposed, the tool's result, the prompt and model version used, and the token cost. The worker's job is short: load the log, work out the next step, execute it, append the result, and release the run.
Resumption is replay, and that is what durable execution means in practice. A worker that picks up a run reads the log and rebuilds the agent's state from recorded results. It does not call the model again for steps already taken. Workflow engines such as Temporal are built on this replay model, where side effects run as activities whose results are recorded in the workflow's history. A plain database table of runs and steps, with a queue and worker leases, gives you the same property if you hold the discipline.
Loading diagram…
Look at the two ways back into Running. One is a human decision and one is a lease timing out, and both resume from the same log by the same replay. There is no separate recovery code path to rot.
Why replay must not ask the model again
Model output is not guaranteed to be reproducible, even at temperature zero, so asking the model a second time for step 14 can return a different plan. Suppose the original step 14 issued a refund and the replayed step 14 decides to email the customer instead. The run's history now describes actions the agent never took, and the refund it did take is invisible to every later step.
That divergence is the failure durable execution exists to prevent. A strong answer says plainly that the model call is a side effect, recorded like any other. A weak answer checkpoints only tool results and assumes it can regenerate the reasoning between them.
Idempotent tools and the crash window
Execution is at least once. A worker can call a tool, have the side effect succeed, and die before writing the result to the log. The next worker will make the same call again.
Close that window with a key derived from the run id and step number, passed to every tool that changes something. The same step always produces the same key, so the retry collides and the tool returns the original result. For a third-party system that does not accept keys, write an intent record before the call, and after a crash read the external state back to find out whether the action happened. Never assume it did not. How a single tool accepts and honours that key is covered in designing tools an agent calls; what the long run adds is that the key must be derived from the step, so a replay on a different worker produces the same one.
Human-in-the-loop approval as state, not as a wait
When the orchestrator classifies a proposed step as risky, it writes an approval request and moves the run to AwaitingApproval. The request holds the resolved arguments, a hash binding the approval to exactly those arguments, the observations the approver needs to judge them, who may approve, and an expiry time. Binding approval to arguments, and placing the gate inside the loop rather than at the start, is covered in stopping an agent from running away. The long-running case adds two problems that a short run never meets.
The first is staleness. An approval granted at 4pm for a step proposed at 10am authorises an action against the world as it was at 10am. Record the versions of the records the step depends on when you raise the request, such as the account balance or the ticket revision, and check them again just before execution. If they changed, the approval is void and the run goes back for a fresh decision.
The second is expiry. Unanswered approvals must time out, and the timeout needs a policy chosen per action: escalate to another approver, stop the run with a partial result, or take a safe alternative the agent already proposed. Approve-on-timeout is never one of the options, because it turns an absent reviewer into consent.
A rejection, or an approver's edit, goes back into the run as an observation with a reason. That lets the agent adapt rather than retry the same step.
Versions, deadlines and the audit trail
Runs outlive deploys, and durable execution makes that the normal case rather than an edge case. Pin the prompt version, model version and tool schema version at run start, and record them on each step. Decide deliberately whether a suspended run resumes on the version it started with or migrates. Both are defensible, but drifting silently between them is not.
Give each run an overall deadline and each step its own timeout. Use worker leases with heartbeats so a lost worker is noticed in seconds, not at the deadline. Hold budgets for tokens and money across the whole run, not per worker session, since a run that resumes ten times must not get ten fresh budgets.
Over hours, the transcript will outgrow the context window. Keep the full step log for audit and replay, and build the model's context from a summary plus the recent steps. Those are two different artefacts, and only the log is the record.
The log is also the audit trail. It shows who approved what, with which arguments, at which step, having seen which observations. Write it append-only and keep it outside the agent's own reach, because a record the agent's tools can edit is not evidence.
Resume from what was recorded, never from what the model would say now; the second answer is a different run.