Your Spark batch pipeline now needs results inside a minute. Do you move to Flink, and what breaks if you do?

Short answer

Do not move a Spark batch pipeline to Flink until the one-minute latency budget is itemized. Check source latency, trigger interval, processing time and sink visibility first; Structured Streaming with short micro-batches often meets a minute. Flink is justified for sub-second latency, per-record state or timer-heavy logic, but it adds long-running state, savepoints, migrations and idempotent sink requirements.

Spend the minute before you spend the migration

"Inside a minute" is a budget, and a budget is useless until it is itemised. Break it into the four places time goes. Put a number on each.

Suppose the requirement is a minute from an event happening to a query returning it. The source has to make the event available: if it is a log-based stream, that is sub-second, and if it is a table polled every five minutes, the floor is already five minutes and nothing downstream can rescue it. Then the processing has to start: an hourly schedule contributes 30 minutes on average and 60 at worst, before any work happens. Then the work itself. Then the result has to become visible to a reader, which is a commit and possibly a metadata refresh.

A plausible allocation looks like a 15-second trigger interval, 20 seconds of processing, and 10 seconds until the write is queryable, which is 45 seconds with 15 to spare. Nothing in that budget requires a different engine. What it requires is that the pipeline stop being scheduled and start being continuous, which in Spark means Structured Streaming with a short trigger rather than a batch job on a cron.

That is the first honest answer, and it is the one most candidates skip past. The default execution model for Structured Streaming is micro-batch, and a micro-batch every few seconds comfortably fits a one-minute end-to-end budget. Moving engines to meet a requirement your current engine already meets is a rewrite you cannot defend at a design review.

When Flink is the right answer

The case for Flink is specific, and stating it precisely is what distinguishes an engineer who has evaluated both from one repeating a comparison they read.

Flink processes records as they arrive rather than accumulating them into a batch, so its latency is not floored by a batch boundary. If your budget is 50 milliseconds rather than a minute, micro-batching cannot get there. That is the clean case.

It also gives you a state model built for the problem. Keyed state with a state backend designed to spill to disk means per-key state larger than memory is a configuration rather than an obstacle. Put a number on it: 10 million keys carrying 200 bytes of state each is 2 GB per parallel instance set, which is uncomfortable to hold on the heap and unremarkable for a disk-backed backend. If your logic is genuinely stateful per entity over a long window, that difference is structural.

And it gives you event time as a first-class concept, with watermarks that make lateness explicit and per-key timers you can set and cancel. Complex per-entity logic - a session that ends after inactivity, a state machine that has to fire after 30 minutes of silence - expresses naturally there and awkwardly in a batch-shaped API.

Exactly-once end-to-end is achievable in both, and it is not the differentiator people think. Both coordinate offsets with output commits; the question in either case is whether your sink participates.

What breaks, which is the part being asked about

The rewrite of your transformation logic is the visible cost. It is also the smallest one. Assume the DataFrame and SQL work does not port, that Flink SQL covers a good deal of it, and that UDFs and connectors are new code. Weeks, not months. It is estimable.

The operational model is where the real cost is, and it is a change in kind rather than in degree.

A batch job starts, does work, and exits. If it fails you rerun it. Its entire state is the input data, so it is trivially reproducible, and the recovery procedure for almost any problem is "run it again". A streaming job is a long-lived process that owns state, and that changes every operational habit you have. It has to checkpoint that state periodically or a failure loses everything since the last one. Restarting is not free - it restores state and rewinds source offsets. Deploying a new version is a savepoint, a stop, a redeploy and a restore, and the new binary has to be able to read the state the old one wrote.

That last clause is the sentence to be able to say. A state schema change that the new job cannot deserialise is a real data-loss event, not a rollback. Your options at that point are to start from empty state and accept a period of wrong output, or to write a state migration. Both are things you plan for weeks in advance, and neither resembles anything in a batch team's experience.

Then the correctness contract changes. Batch reprocessing is idempotent because you overwrite a partition: run it twice, get the same partition. Streaming output is incremental, so writing twice means duplicating unless the sink is transactional or the write is keyed and upserting. "Just rerun it" stops being available as a universal repair, and every fix has to reason about what has already been emitted.

Backfilling deserves its own paragraph because it is routinely discovered late. Reprocessing two years of history through a streaming topology means feeding it historical data faster than real time while its watermarks, timers and windows behave sensibly, and a pipeline written for live traffic frequently does not survive that. Teams end up maintaining a batch path for history and a streaming path for the present, computing the same numbers two ways, and then reconciling them. That duplication is a permanent cost of the move and belongs in the estimate.

The question to ask before any of this

Where does the data come from, and how quickly does the source make an event available?

If the answer is a nightly extract, a message queue polled on a schedule, or a database with a batch export, then the latency floor is upstream of everything you were about to design, and no engine choice reaches one minute. The work is to get change data out of that source continuously, which is a different project with different owners. That is often the whole answer to the requirement, and it involves no engine change at all.

The mirror image is the sink. If the results land somewhere whose readers see a snapshot refreshed every ten minutes, then a pipeline producing output in two seconds still delivers a ten-minute answer. Both ends of the budget are easy to overlook precisely because the interesting engineering is in the middle.

The trap is answering the engine question first

The requirement arrives phrased as a technology comparison, and the phrasing invites you to compare technologies. An interviewer asking this is usually checking whether you interrogate the requirement before satisfying it.

So the strong answer opens by itemising the minute, establishes the source floor, tries the cheaper change - a short trigger interval on the engine already in production - and reaches for Flink when the budget is genuinely sub-second or the logic is genuinely per-record and stateful. Then it prices the move in operational terms rather than in lines of code, because the savepoint, the state migration and the backfill path are what the team will live with.

There is also a middle answer worth naming. Keep the batch pipeline for correctness and history, add a streaming path for the fresh slice, and serve queries from both. It costs you a reconciliation obligation and it lets you ship a one-minute answer without betting the whole pipeline on a new engine. Whether that is pragmatic or an accumulation of debt depends on how long you intend to run both, and saying so directly reads better than pretending either extreme is free.

Itemise the minute before choosing an engine, because a short trigger interval on the engine you already run usually fits it. Flink's cost is not the rewrite, it is trading a rerunnable job for a long-lived one that owns state you have to migrate.

---

© 2026 Preptima. Originally published at preptima.com.