hardDesignScenarioSeniorStaffLead
How would you cut the cost of a GPU training fleet without slowing the team down?
6 min readUpdated 2026-08-15
Answer
Short answer
Measure utilisation first, because most fleets pay for idle accelerators rather than for too few. Then right-size to whatever each job is bound by, move interruptible work onto spot capacity behind tested checkpointing, and pool the fleet behind one queue instead of reserving per team.
Was this answer helpful?
Likely follow-ups
- Your average GPU utilisation is 18%. What are the three most likely causes?
- How would you set the checkpoint interval, and what does checkpointing itself cost you?
- A team asks for dedicated GPUs so their jobs never queue. How do you respond?
- Which jobs would you refuse to run on spot capacity, and why?
Related questions
- A training job dies eight hours in with nothing to show for it. What should have been in place?mediumAlso on checkpointing and spot-instances5 min
- How would you autoscale a GPU inference service?hardAlso on gpu6 min
- How do you decide whether to use a managed service or self-host a component, and which cloud costs catch teams out?hardAlso on cost-optimisation6 min
- You serve millions of prompts an hour and the LLM bill doubled. Where does the money go, and what do you cut?hardAlso on cost-optimisation5 min