Batch, online and streaming inference, latency budgets, autoscaling and cold starts, GPU sharing, and shadow deployments.
3 questions in MLOps & ML Platform.
Freshness decides it. If a prediction is still valid hours later, precompute it in a batch job and serve a lookup; if it depends on what the user did seconds ago, score it online; if it depends on an evolving event sequence, score it from a stream. Most mature systems end up hybrid.
Scale on queue depth or in-flight concurrency rather than request rate, because a GPU replica that takes minutes to load weights cannot be added after the queue has already formed. Then decide deliberately how much warm capacity to keep, since that is what you are paying to avoid cold starts.
Break the budget into its parts first, because feature fetching usually costs more than the model does. Then parallelise and cache the fetches, shrink the model with quantisation or distillation, set a timeout shorter than the budget, and define a degraded path that answers without the model.