GPU economics, spot and preemptible capacity, right-sizing inference, caching, and knowing the unit cost of a prediction.
3 questions in MLOps & ML Platform.
Divide the all-in hourly cost of the serving capacity by the predictions it delivers in that hour at your real utilisation, then add the per-request cost of feature reads. Utilisation dominates the result, and most of what people call cost per prediction is fixed cost that one more request does not change.
Measure utilisation first, because most fleets pay for idle accelerators rather than for too few. Then right-size to whatever each job is bound by, move interruptible work onto spot capacity behind tested checkpointing, and pool the fleet behind one queue instead of reserving per team.
Flat traffic and a tripled bill means cost per request tripled, so hunt what changed per request: a larger model version, more feature reads per prediction, retries multiplying calls, or warm capacity left running. Per-request cost attribution is what makes this answerable.