mediumConceptDesignMidSeniorStaff
How do you decide whether a model should be served in batch, online or streaming?
4 min readUpdated 2026-08-15
Answer
Short answer
Freshness decides it.If a prediction is still valid hours later, precompute it in a batch job and serve a lookup; if it depends on what the user did seconds ago, score it online; if it depends on an evolving event sequence, score it from a stream.
Was this answer helpful?
Likely follow-ups
- Your batch job scores every customer nightly but only 4% of them log in that day. What would you change?
- A product manager asks for real-time recommendations. How would you find out whether they need real time or just fresh?
- What breaks first when you take a model written for batch scoring and put it in a request path?
- How would you serve a model whose features need a 30-day aggregate and a 30-second one?
Related questions
- Users say the LLM feature in your product feels slow. How do you work out what to fix?mediumAlso on latency and streaming5 min
- What does batching buy you when serving an LLM, and what does it do to your latency budget?hardAlso on latency and inference5 min
- You have 50 ms for a model call in a request path. How do you make that budget?hardAlso on latency and inference5 min
- What is the KV cache, and why does it decide how many users one GPU can serve?hardAlso on inference4 min