Top 14 Streaming Interview Questions
Browse 14 real-world technical and behavioral interview questions about Streaming. Review scenarios, edge cases, and architectural best practices.
Design a chat service that streams model output to users and lets them press stop. What has to happen when they do?
Stream tokens over server-sent events or a WebSocket, but treat stop as a server-side command routed to whichever worker owns the generation. That worker aborts the upstream model call, saves the partial reply with a stopped status, and sends a final event so the client and the stored history agree.
Rate-limit an LLM gateway where one request can cost a hundred times another.
Rate limiting an LLM gateway by request count fails because cost is tokens, and output tokens are unknown at admission. Budget tokens per tenant, reserve input plus expected output, settle against reported usage when the stream ends, cap concurrent streams, and cancel upstream on disconnect.
How do you decide whether a model should be served in batch, online or streaming?
Freshness decides it. If a prediction is still valid hours later, precompute it in a batch job and serve a lookup; if it depends on what the user did seconds ago, score it online; if it depends on an evolving event sequence, score it from a stream. Most mature systems end up hybrid.
The same domain has to be exposed to a mobile app, a partner integration and internal service-to-service traffic. Where does GraphQL fit, where does gRPC, and where does neither?
Choose per consumer: GraphQL where many client shapes make over-fetching the problem, gRPC for internal traffic wanting typed generated stubs and streaming, plain JSON over HTTP for partners who want curl and a CDN. Then batch your resolvers, because the N+1 is structural.
Users say the LLM feature in your product feels slow. How do you work out what to fix?
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
How do you achieve true exactly-once semantics in Flink across source, state, and sink without cratering throughput?
An analysis of the architectural reality of exactly-once guarantees in Flink, from distributed snapshots to two-phase commits. Use this data engineering answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects streaming to the point an interviewer is testing.
You move an internal service to gRPC behind the same load balancer your REST services use, and one pod ends up serving most of the traffic. What is happening, and what do you change?
gRPC multiplexes many calls over one long-lived HTTP/2 connection, so a balancer that distributes connections pins each client to one backend for its lifetime. Balance per request with an HTTP/2-aware proxy or client-side subchannels, and bound connection age so scaling out actually moves traffic.
You are handed k sorted lists and asked for one sorted list. What is wrong with concatenating and sorting?
Nothing about correctness - it works. It throws away the sortedness you were given and pays O of N log N to rebuild it, where merging costs O of N log k, and it requires every element resident at once where a merge requires k. On lists that arrive as streams the second objection is the one that disqualifies it outright.
Numbers arrive one at a time and after each one you must answer what the Kth largest so far is. What do you keep, and what do you deliberately throw away?
Keep a min-heap holding exactly the K largest values seen. Its root is the answer, so the query is a read rather than a search, and every value smaller than that root is discarded the moment it arrives. What you give up is the ability to answer for any K larger than the one you chose, which is why the first thing to establish is whether K is fixed.
Prices must react to demand within seconds, and two users looking at the same item must not see different prices. Which do you give up?
Real-time pricing system design should publish numbered price versions instead of calculating a different answer per request. Compute prices by region and time window, serve the current version consistently, and freeze quotes briefly so users are not surprised during checkout.
How do you architect a strictly low-latency, real-time pipeline to ingest, embed, and index tens of thousands of unstructured documents per second into a vector database?
Designing a robust streaming architecture for embedding generation, detailing GPU adaptive batching, Kafka-driven backpressure management, and asynchronous HNSW indexing. Use this machine learning answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects data engineering to the point an interviewer is testing.
Numbers arrive one at a time and after each one I want the running median. How would you keep it?
Split the values across a max-heap of the lower half and a min-heap of the upper half, so the median is read off one or both tops. Balancing by size alone is what breaks it: a new value has to enter through the heap it belongs to by value and be transferred across, never pushed into whichever heap is currently smaller.
Return the k most frequent elements from a large stream of values. Why not just sort by frequency?
Count occurrences, then push counts through a min-heap capped at size k, evicting the smallest whenever the heap overflows. That is O(n log k) rather than the O(n log n) of a full sort, it needs only k items resident at once, and it works on a stream you cannot re-read.
You stream the model's answer token by token to the browser. What does that change about error handling, guardrails and cost?
Streaming commits you to the response before you have seen it: the status code is already sent, any check that needs the whole output now runs after the user has read part of it, a mid-stream failure has no clean retry, and an abandoned tab keeps generating tokens you pay for. It also connects llm serving to the point an interviewer is testing.