The monthly bill run dies two thirds of the way through and the cycle closes tomorrow. What do you do?

First establish what has already escaped

Before restarting anything, find out what the completed portion did to the outside world. An issued invoice is not a row you can reconsider. It has been posted to the receivables ledger, rendered as a document the customer can already open in their app, quite possibly emailed, and it may have instructed a direct debit that will be presented to a bank on a date nobody can recall.

That inventory of side effects is the whole basis for what follows, because it tells you that a bill run is not a transaction and cannot be treated as one. It is a long sequence of individually irreversible commitments, and the only useful question is which of them completed.

The unit of atomicity is the account, not the run

A run over a cycle group is resumable only if each account is in one of two states: fully billed for this period, or untouched by this period. Half-billed accounts are where recovery becomes guesswork, so the design has to make them rare and identifiable.

That means durable per-account progress recorded as the run proceeds, keyed on the account, the cycle and the period, so a second execution can skip what is done. It means the sequence of commit points inside one account is enumerated and ordered deliberately — charges assembled, invoice persisted, number allocated, document rendered, ledger posted, collection instructed, notification sent — with the later steps re-drivable from the persisted invoice rather than from the charge assembly. Re-assembling charges for an account whose invoice already exists is how you produce a second, slightly different invoice for one period.

Invoice numbering deserves specific mention because it is where naive retries show up in an audit. Numbers usually have to be gapless and single-use for tax purposes, so allocation must be durable and tied to the account and period, not taken from a counter incremented at the moment of rendering. Otherwise the retry burns a number and the sequence has a hole nobody can explain.

Restart, do not rerun

The distinction is the answer to the question as asked. Rerunning the job from the beginning re-bills every account that already succeeded, which duplicates real invoices, doubles the receivables and presents the same direct debit twice — a materially worse outcome than the original failure, and one that reaches the regulator rather than the support queue.

Restarting means re-invoking the same run, identified by the same cycle and period, and having it converge: skip the accounts already complete, finish the ones part way through from their last commit point, and process the remainder. The idempotency key is the account plus the cycle plus the period. It is the same principle as an idempotent provisioning task, applied to something that moves money.

Restarting only works if the inputs were frozen

Here is the property that decides whether the restart produces the right numbers, and it has to have been designed in weeks earlier.

The accounts billed before the failure were priced against the catalogue, the subscription state and the tariff versions as they stood when the run began. If the restart resolves those live and anything has changed in between — a price update released overnight, a retention discount applied by an agent, a subscription cancelled this morning — then two subsets of one cycle group are billed on two different bases for the same period. The totals will not reconcile, the difference will be small enough to survive review, and it will surface months later as a revenue-assurance finding nobody can attribute.

So the run must operate on inputs pinned at cycle cut-off: the catalogue version, the effective-dated subscription timeline as at that instant, the set of rated usage admitted to the period. Charge assembly then becomes a deterministic function of stored inputs, and a restart six hours or three days later produces byte-identical figures. This is the same versioning discipline that lets you defend a single invoice line in a dispute, and it is what makes recovery boring.

Diagnose before you restart

A restart that drives straight back into the cause fails at the same place and burns hours you do not have. The classification matters more than the speed.

If the failure is infrastructural — a database connection lost, a node evicted, disk exhausted — restart once the cause is cleared and expect it to complete. If the failure is data, restarting is pointless, because the account that produced a null where the code expected a value will produce it again. Data failures are the common case, and they are usually concentrated: a handful of accounts with an unusual combination such as a mid-cycle migration, a bundle nobody has sold since, or a credit note referencing an invoice from a system that has been decommissioned.

For those, quarantine and continue. Move the offending accounts out of the run into an exception set, bill the remaining population, and handle the exceptions as a small, named, tracked list. Ninety accounts unbilled at cut-off is an exception report with owners; a whole cycle group unbilled is a delayed revenue recognition, a spike in the collections cycle, and a conversation with finance about the month-end close. Choosing the manageable failure over the tidy one is the judgement being tested.

The shortcuts to refuse under deadline pressure

Two suggestions will be made in the room, both by someone reasonable, and both should be declined.

The first is to quietly extend the period by a day so the run has time. That moves the period boundary, so usage from the extra day is billed in this period and every downstream comparison against the previous month is now against a longer month. It also changes what customers were told their billing period was.

The second is to issue estimated invoices for the unbilled remainder and correct them next cycle. An estimate you intend to correct is still a document with a legal character that the customer holds, and correcting it produces two adjustments per account against a figure that was never right, for a population large enough that the correction becomes its own project.

What a run should publish so this is provable

Recovery is easier when the run is instrumented as a control rather than as a job. Every execution should emit reconcilable counts: accounts in scope for the cycle group, accounts billed, accounts quarantined with a reason, accounts skipped as already complete, and the value totals alongside them. Those numbers must add up to the population, and the check that they add up should be automatic.

That is what lets you answer the question the CFO asks after an incident, which is never "what broke" but "are you certain every customer got exactly one invoice for June". Without those counts the honest answer is that you believe so, and belief is not what is being asked for.