How do you achieve true exactly-once semantics in Flink across source, state, and sink without cratering throughput?
An analysis of the architectural reality of exactly-once guarantees in Flink, from distributed snapshots to two-phase commits. Use this data engineering answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects streaming to the point an interviewer is testing.
What the interviewer is scoring
- Whether they understand the coordination between Flink's checkpointing and a two-phase commit sink.
- Does the candidate diagnose and resolve checkpointing bottlenecks using incremental RocksDB snapshots?
- That they recognise the impact of bounded out-of-orderness watermarks on state size and latency.
- Whether the candidate can explain the Chandy-Lamport distributed snapshot algorithm in a streaming context.
- Whether they factor in network partitions and task manager failures when designing the recovery strategy.
Answer
Short answer
An analysis of the architectural reality of exactly-once guarantees in Flink, from distributed snapshots to two-phase commits.
The illusion of internal guarantees
Stream processing at scale demands exactly-once semantics, but naive implementations often mistake Flink's internal guarantees for end-to-end data integrity. A high-throughput pipeline consuming from Kafka, maintaining windowed aggregations, and writing to a relational database like PostgreSQL cannot rely solely on the stream processor's promises. If the sink uses default auto-commit, any failure after a flush but before a checkpoint will result in duplicated writes upon recovery. The true transactional boundary must encompass the source offsets, the distributed state, and the final sink.
The false comfort of checkpointing alone
The naive approach assumes that enabling Flink's checkpointing automatically bestows exactly-once semantics across the entire pipeline. Engineers configure an in-memory state backend, write to a generic JDBC connector, and assume the framework handles the rest. Under peak load, this facade crumbles. The in-memory state outgrows available heap, checkpointing durations spike, and barriers timeout. The job restarts, drops in-flight transactions, and duplicates data at the sink, violating the core requirement.
Coordinating the distributed snapshot
Achieving actual exactly-once semantics requires orchestrating a two-phase commit (2PC) protocol that aligns with Flink's implementation of the Chandy-Lamport algorithm. The source (Kafka) must manage offsets transactionally, integrating deeply with the checkpoint coordinator. The sink (PostgreSQL) must only commit transactions when a checkpoint successfully completes. If a failure occurs before the barrier arrives, the uncommitted database transaction aborts, Flink rolls back its state to the prior snapshot, and Kafka replays the offsets. This transforms a sequence of independent operations into a cohesive, atomic state transition.
Tuning state and managing chaos
Long-running transactions hold locks in relational sinks, risking split-brain scenarios if task managers fail and lose track of in-flight commits. Transaction timeouts must be aggressively tuned against the checkpointing interval. Furthermore, massive windowed aggregations will choke an in-memory state backend. Migrating to RocksDB enables incremental checkpoints, offloading only the delta to the object store and drastically reducing barrier alignment times. RocksDB itself demands rigorous tuning of block caches and write buffers to prevent silent out-of-memory kills at the task manager level.
The reality of out-of-order data
Late-arriving events inevitably complicate exactly-once guarantees. Relying on strict chronological processing in a distributed system is architectural suicide. Bounded out-of-orderness watermarks balance latency against completeness. Events arriving after the watermark closes the window must be decisively routed to a side output, ensuring they do not corrupt the primary aggregations. At extreme scale, this requires an exact science of balancing eviction strategies, watermark advancement, and state time-to-live (TTL) to prevent unbound memory growth while preserving strict accuracy.
flowchart TD
A["Kafka Source (Offsets)"] --> B["Window Aggregation Operator"]
B --> C["RocksDB State Backend"]
B --> D["PostgreSQL Sink (2PC)"]
E["Checkpoint Coordinator"] -.-> A
E -.-> B
E -.-> D
C -.-> F["Object Store (Incremental Snapshots)"]The true challenge in exactly-once semantics lies not just within the stream processor, but in maintaining a coherent transactional boundary across the source, the processor's state, and the sink, while simultaneously optimising the state backend to prevent checkpointing bottlenecks.
© 2026 Preptima. Originally published at preptima.com.
Likely follow-ups
- How do you handle a sink that has no transactional semantics at all, such as a plain REST API?
- What happens to your exactly-once guarantees when you need to rescale a stateful operator's parallelism?
- How would you detect in production that your two-phase commit sink is silently duplicating records?
Related questions
- How do you architect a strictly low-latency, real-time pipeline to ingest, embed, and index tens of thousands of unstructured documents per second into a vector database?hardAlso on data-engineering and streaming3 min
- How would you design a data lakehouse to handle petabytes of data with frequent GDPR right-to-be-forgotten requests and rapid schema evolution without sacrificing query latency?hardAlso on data-engineering3 min
- How do you implement dynamic PII masking across a highly decentralised data mesh without destroying the analytical utility of the data for downstream machine learning workloads?hardAlso on data-engineering2 min
- Your Spark batch pipeline now needs results inside a minute. Do you move to Flink, and what breaks if you do?hardAlso on flink6 min