Top 6 Partitioning Interview Questions
Browse 6 real-world technical and behavioral interview questions about Partitioning. Review scenarios, edge cases, and architectural best practices.
A transform has been writing wrong revenue figures for three days and six downstream tables have consumed it. How do you backfill the corrected data without double-counting anything?
Make the write idempotent before you make it correct: partition-level replace rather than append, so rerunning a day produces the same result whether it runs once or five times. Then work out how far the contamination spread, because a downstream table that accumulated rather than recomputed will not fix itself when the source is repaired.
Consumer lag on your orders topic climbs every morning and never clears before the next peak. Work through it.
Lag is the integral of arrival rate minus service rate, so establish which of the two moved and whether the consumer is slow or stalled. Check lag per partition before per group, rule out a rebalance loop, and remember that adding consumers beyond the partition count does nothing.
Every message must survive the phone being wiped, and a user has 40,000 of them. Where do they live and how are they paged?
The server holds one row per message, partitioned by conversation and clustered by descending sequence so a page is a contiguous read, and clients page by keyset cursor rather than offset. Per-user state moves to a side table so the body stays shared, and if messages are end-to-end encrypted the wipe is a key-custody question, not a storage one.
How do you choose a datastore, and then how do you choose its shard key?
Choose the store from the dominant access pattern and the invariants it must enforce, not from a category label. Choose the shard key so the most frequent query is answerable from one shard, the key has enough distinct high-traffic values to spread load, and shards can be split later without rewriting every row.
Your nightly Spark job writes tens of thousands of tiny files, and the job that reads them the next morning is slow. What happened, and what do you change?
The file count is the number of writing tasks multiplied by the partition values each task holds, so a wide shuffle plus a high-cardinality partition column produces a file explosion. Repartition to match the write layout rather than coalescing to one task, and compact behind a table format.
A graph of two billion edges will not fit on one machine and the algorithm needs neighbours. How do you partition it?
Check the premise first, because two billion edges as vertex id pairs is about 32GB and that fits in one large machine. If it genuinely must be distributed, partition edges rather than vertices, since real graphs are power-law and a single high-degree vertex under an edge cut makes its machine the straggler that every synchronous superstep waits for.