How would you cut the cost of a GPU training fleet without slowing the team down?

Measure utilisation before buying or cutting anything

The first question is not which instances to change, it is how much of what you already have is doing work. Two numbers, and they are usually far apart: allocated GPU-hours, meaning accelerators reserved by a job, and busy GPU-hours, meaning accelerators executing kernels. The ratio is your real utilisation and it is the single most informative figure in the conversation.

Do the arithmetic on why it matters. A fleet of eight accelerators running at twenty per cent average utilisation is delivering the work of 1.6 accelerators. If you could raise that to sixty per cent you would do the same work on roughly three, and no negotiation over instance types comes close to that. Cost optimisation on a training fleet is almost always a utilisation problem wearing a procurement costume.

Utilisation is lost in three distinguishable places, and they have different fixes. Accelerators allocated to a job that is stalled waiting for data. Accelerators allocated to nothing at all, because a team holds a reservation for jobs they run twice a week. And accelerators allocated to a job that finished, or crashed, and whose notebook or process is still holding the device — which is more common than anyone admits and is worth an automated reaper on its own.

Right-sizing means finding out what the job is bound by

Profile before resizing. A training step is bound by one of a small number of things and the correct action differs completely per case.

If the accelerator is idle a large fraction of each step, the job is input-bound: the data loader, the decode, the augmentation or the object-store read cannot keep up. Adding accelerators here makes the problem worse and the bill larger, because each new device waits for the same starved pipeline. The fix is more loader workers, prefetching, a cheaper data format, or local caching of the dataset.

If the device is busy but memory is barely used, the job is running on more accelerator than it needs, and either a smaller device or a larger batch size — which converts unused memory into throughput — is the answer. If memory is full and the device is busy, you are correctly sized and the remaining levers are algorithmic: mixed precision, gradient accumulation instead of more devices, activation checkpointing to trade compute for memory.

If a multi-device job scales badly — doubling devices gives well under double the throughput — you are paying for communication, and adding devices past that point buys almost nothing. Knowing where a job's scaling curve flattens is what lets you say no to a request for sixteen accelerators for a job that stops improving at four.

Spot capacity, and the checkpointing that makes it safe

Interruptible capacity is the largest single price lever available for training, because training is the rare workload that can genuinely tolerate being killed. It is only true if checkpointing works, so treat checkpointing as the precondition rather than as a detail.

The economics are easy to state with your own numbers. Suppose interruptible capacity costs a fraction f of on-demand for the same device, and that interruptions cause you to lose some fraction of the work you pay for. Take f as 0.4 and assume you lose ten per cent of paid time to work that was redone after an interruption. Your effective cost is 0.4 divided by 0.9, about 0.44 of on-demand — so most of the headline discount survives, and the redone work is the tax. That framing also shows what to optimise: the discount is fixed by the market, but the ten per cent is yours to reduce, and it is a function of your checkpoint interval.

For the interval itself, the expected lost work per interruption is roughly half the interval, since an interruption falls at a random point between checkpoints. So if interruptions arrive about every two hours and you checkpoint every thirty minutes, you expect to lose about fifteen minutes per interruption, or roughly an eighth of the time between them. Halving the interval halves the loss but doubles the checkpoint overhead, and checkpointing a large model is not free — it serialises optimiser state as well as weights, which is several times the parameter size, and writes it to storage. Measure how long a checkpoint takes and set the interval where the two costs balance rather than picking a round number.

Two details make the difference between checkpointing that works and checkpointing that appears to. The state you save must include everything needed to resume identically: model weights, optimiser state, learning-rate schedule position, data-loader position and the random-number generator state. Miss the data-loader position and a resumed run silently re-reads the same examples, which changes the training distribution. And writes must be atomic — write to a temporary object and rename, because a checkpoint interrupted mid-write is worse than no checkpoint at all when it is the newest one on disk.

Keep some capacity non-interruptible. The final production training run before a release, anything with a deadline attached, and any job whose checkpointing you have not tested belong on stable capacity. A mixed fleet where exploratory work is interruptible and release-critical work is not gets most of the saving without the risk.

Sharing the fleet is what keeps the team fast

Per-team reservations are the most expensive way to make people feel safe. Each team sizes its reservation for its own peak, peaks do not coincide, and the fleet-wide average utilisation is therefore the average of several individually low numbers. Pooling the same accelerators behind one queue lets one team's trough serve another's peak, which is the same argument as any capacity-sharing argument and it works here for the same reason.

The trade the team cares about is queue wait, so make it explicit. Give each team a guaranteed share it can always claim, allow borrowing above that share when the fleet is idle, and make borrowed capacity preemptible so a guarantee is honoured within a bounded time. Add priority classes so a release-blocking run jumps an exploratory sweep. Gang-schedule multi-device jobs so a four-device job does not sit half-allocated holding two accelerators hostage while it waits for the other two — that specific failure both wastes capacity and lengthens the queue, which is the worst of both.

Then measure queue wait at the p90 and treat it as the constraint you must not regress. "Cut cost without slowing the team down" has a metric, and this is it. Cost per useful training hour going down while p90 queue wait stays flat is the result you are claiming; cost going down while queue wait triples is a cut, not an optimisation, and someone will notice within a fortnight.

The savings that need no negotiation

Three things are pure waste and cost nothing politically to remove. Idle-timeout every interactive session and development notebook, because a device held by a dormant session is the cleanest waste in any fleet. Cap and expire hyperparameter sweeps with early stopping, so a sweep with an obviously losing configuration does not run to completion twenty times. And cache derived datasets rather than recomputing the same preprocessing at the start of every run, which shows up as accelerator-idle time in the profile and as duplicated compute in the bill.

The number to establish before anything else is busy GPU-hours over allocated GPU-hours. Almost every credible saving on a training fleet is a way of raising that ratio, and the metric proving you did not slow the team down is p90 queue wait.