A training job dies eight hours in with nothing to show for it. What should have been in place?

What a checkpoint has to contain

The first mistake is checkpointing the model weights alone. Restart from weights and you resume with a freshly initialised optimiser, which for adaptive optimisers means losing the accumulated moment estimates that were shaping every update. The loss visibly jumps and the run behaves as though it has been partially reset, because it has.

A resumable checkpoint holds the model parameters, the optimiser state, the learning-rate scheduler's position, the step and epoch counters, the state of every random number generator involved, and the position of the data loader within the current epoch. It also holds the gradient-scaler state if you are training in mixed precision. Omit the data-loader position and the resumed run re-reads examples it has already seen while never reaching the tail of the epoch, which is a silent change to the training distribution rather than a crash.

The RNG state deserves a note because it is easy to reason about wrongly. Restoring it makes the resumed run's augmentation and shuffling continue the sequence rather than restart it. Without that, resume is not equivalent to continuing, and a run that was interrupted twice has seen a different data ordering than one that was not. Whether this matters depends on how much you care about run-to-run comparability, but you should know that it is a difference and not assume the resume is exact.

Writing them so they survive the failure

A checkpoint written to the instance's local disk is not a checkpoint if the failure mode you are protecting against destroys the instance. Spot reclamation, node preemption and hardware failure all do. The checkpoint has to land in durable storage outside the machine, which introduces a write duration you have to plan for.

The write must be atomic. If the process dies partway through serialising the newest checkpoint, and it was writing over the only copy you had, you have converted a recoverable failure into a total one. Write to a temporary name and rename or commit as the last step, and keep a small rolling set of recent checkpoints rather than one, so a corrupt newest file costs you one interval rather than the whole run. Keeping the best checkpoint by validation metric separately from the most recent one is worth doing too, since they answer different questions: the latest is for resuming, the best is for shipping.

# Save under a temporary path, then commit by rename. A crash mid-write
# leaves the previous good checkpoint intact.
tmp = f"{ckpt_dir}/step-{step}.tmp"
save({
    "model": model.state_dict(),
    "optimizer": optimizer.state_dict(),   # omit this and the loss jumps on resume
    "scheduler": scheduler.state_dict(),
    "step": step,
    "epoch": epoch,
    "sampler_position": loader_state,      # omit and the epoch is silently reshuffled
    "rng": capture_rng_states(),
}, tmp)
commit(tmp, f"{ckpt_dir}/step-{step}.ckpt")
prune_all_but_newest(ckpt_dir, keep=3)

Choosing the interval from expected cost

The interval is an optimisation, and you should show the arithmetic rather than asserting a number. Two quantities set it: the cost of a checkpoint write, and the expected amount of work at risk between writes.

Suppose a checkpoint takes two minutes to serialise and upload, and you expect roughly one interruption per twelve hours of training. Checkpoint every ten minutes and you spend twenty per cent of the run writing checkpoints, which is absurd. Checkpoint every four hours and each interruption loses, on average, two hours of accelerator time. Somewhere between those the marginal cost of another write equals the marginal expected saving, and with a two-minute write and hourly-scale interruption rates that lands in the region of once or twice an hour. The point of the calculation is not the answer, which depends entirely on your write cost and your failure rate; it is that both numbers are measurable and neither is a matter of taste.

Two refinements are worth mentioning. If the write can be overlapped with computation — snapshotting state to host memory quickly and uploading asynchronously — the effective cost of a write drops sharply and the optimal interval shortens with it. And on very large jobs the checkpoint itself becomes large enough that the storage bandwidth, not the training, sets the floor.

Interruption handling, which is different from resumability

Spot and preemptible capacity is typically offered at a steep discount against on-demand, and the provider normally gives a short notice before reclaiming the machine. That notice is the whole design opportunity. A job that is merely resumable loses everything since its last periodic checkpoint. A job that handles the notice — catches the termination signal, writes a checkpoint immediately, and exits cleanly — loses almost nothing, which changes the economics of using discounted capacity at all.

preemption notice received
      |
      v
stop after current step  ->  write checkpoint  ->  exit non-zero
      |
      v
orchestrator reschedules  ->  new node  ->  resume from checkpoint

The constraint is that the notice window is short and fixed, so the emergency checkpoint has to fit inside it. If your normal checkpoint takes four minutes and the notice is two, the handler is decoration. That is a real design input: it may push you to keep checkpoints smaller, to shard them across workers, or to accept periodic checkpointing alone on the largest jobs.

Around this you need the orchestration to cooperate. The retry must land on a fresh node, must pass the checkpoint directory through, and must not count preemptions against a retry budget intended for genuine errors — otherwise a run on cheap capacity exhausts its retries and fails for a reason that was expected. And you want a distinction in the exit handling between "interrupted, resume me" and "the data was bad, stop", because retrying the second wastes another eight hours reaching the same conclusion.

Where this answer usually falls short

Candidates say "we checkpoint" and stop, and the interviewer is listening for whether resume has ever been exercised. Checkpointing is the easy half; the failure that costs a team a week is discovering, at hour eight of a rerun, that the resume path throws on a key mismatch, or silently starts from step zero, or restores weights and quietly drops the optimiser. Untested resume code is a plan, not a capability.

Make it real by resuming deliberately and often. Kill a short run halfway and confirm the resumed loss curve continues rather than jumps. Have the pipeline check for an existing checkpoint on every start, so resume is the normal code path taken by every run rather than an emergency branch nobody has executed. And keep an eye on liveness while you are at it: a job that has stopped making progress but is still holding accelerators is more expensive than one that crashed, because nothing tells you.

Checkpointing without a resume path you have deliberately exercised is a false sense of safety. The interruption signal is the cheapest thing in this design to handle and the thing that makes discounted capacity viable at all.