How do you decide whether a model should be served in batch, online or streaming?

Freshness is the variable, not the technology

Ask how stale a prediction is allowed to be before it stops being useful, and insist on an answer in units of time. That number decides the serving mode; everything else is consequence.

A churn propensity computed at 03:00 is still a defensible churn propensity at 17:00, because the behaviour it summarises moves over weeks. A fraud score for the payment being authorised right now is worthless if it was computed last night, because the signal that matters is what this card did in the previous ninety seconds. A delivery-time estimate sits in between: it depends on conditions that change over minutes, so it must be recomputed as events arrive but not necessarily inside the customer's request.

That gives three modes with three different contracts. Batch precomputes scores for a known population on a schedule and writes them somewhere cheap to read. Online computes a score inside a request, for whatever entity the request names, from data available at that instant. Streaming computes continuously as events land, keeping a running state per entity and emitting or updating a score whenever the state changes materially.

The cost profiles differ in shape, not only in size

Work through it with your own numbers rather than asserting that one is dearer. Suppose you must score ten million customers a day and the model takes two milliseconds of CPU per row. That is twenty thousand CPU-seconds, or roughly 5.6 CPU-hours of actual compute. Run as a batch job on a sixteen-core machine, that is about twenty-one minutes of wall clock on one machine once a day, and you pay for twenty-one minutes.

Now serve the same ten million scores online, one per request. The model compute is unchanged — still 5.6 CPU-hours consumed — but you no longer get to choose when it happens. If traffic peaks at four times the daily mean and you hold thirty per cent headroom above peak so that autoscaling has time to react, you are provisioning for roughly five times your mean load. You pay for something like five times the compute you consume, plus the feature-fetch infrastructure that batch did not need because it read the warehouse directly.

Streaming has a third shape: a consumer group and its state store run continuously whether or not anything interesting is happening, so the floor is a fixed always-on cost, and the marginal cost per event is small. That makes it good value at high event volume and poor value for a topic that is quiet for twenty hours a day.

The hybrid most production systems converge on

The interesting design is rarely one of the three. It is a batch-computed base, kept in an online store, refreshed or adjusted online by the few features that genuinely cannot wait.

Recommendations are the canonical case. Candidate generation and item embeddings are computed nightly, because the catalogue and the long-run affinities do not change per request. The request path fetches that precomputed candidate set and re-ranks it using session features — what this user has viewed in the last two minutes — which is a small model over a small input and fits a tight budget. Fraud works the same way in reverse: the thirty-day counters are precomputed and materialised to the online store, and the request path computes only the handful of features derived from the current session before calling the model.

flowchart LR
  A[Nightly batch job] --> B[Precomputed scores<br/>and aggregates]
  C[Request arrives] --> D[Fetch precomputed values]
  B --> D
  C --> E[Compute session features<br/>from recent events]
  D --> F[Model call]
  E --> F
  F --> G[Response]

Note where the two paths join: the request never recomputes the thirty-day aggregate, so the expensive history is paid for once a night rather than once per request.

Why answering with latency instead of freshness goes wrong

A candidate asked this question very often answers with the response-time requirement — "we need to reply in fifty milliseconds, so it has to be online". That reasoning is backwards, and it is the single clearest separator in this question.

Batch serving is the fastest thing at read time. A key lookup of a precomputed score in a low-latency store beats any online model call, because no model runs. A tight latency budget is therefore an argument against computing something expensive in the request path, which is an argument for precomputation wherever the score permits it. The question online inference answers is different: does the prediction depend on information that did not exist when the batch job ran? If it does, no amount of precomputation helps. If it does not, online inference is buying you nothing and charging you peak-provisioned compute for it.

The mirror-image error is choosing streaming because there is already a message broker in the architecture. Streaming earns its always-on cost when the prediction depends on an ordered sequence of events and you need the score updated before anyone asks for it — a session that becomes suspicious after the fourth action, an asset whose sensor trace crosses a threshold. If the score is only ever read in response to a user request, an online model reading fresh features is simpler and cheaper than maintaining stream state.

Decide with one number: how old may this prediction be? Everything after that — where the compute happens, what you provision for, whether you need stream state — follows from it.