Spark & Big Data interview questions

Partitioning and shuffles, skew, joins at scale, catalyst and lazy evaluation, and cluster tuning.

4 questions in Data & AI Engineering.

Data and AIA 22-minute read on how the topic fits together, before the questions.Data and AI engineering fundamentals58 short answers on one page, for revising rather than studying.

hard4

hardDesignScenario

Your Spark batch pipeline now needs results inside a minute. Do you move to Flink, and what breaks if you do?

Usually not yet. Itemise the minute across source, trigger interval, processing and read visibility, because a few-second micro-batch in Structured Streaming often fits it. Flink earns the move at sub-second latency or per-record state, and what breaks is the operational model: a long-lived job whose state you must migrate, and a sink that must be idempotent.

6 minmid, senior, staff

hardScenarioConcept

Your nightly Spark job writes tens of thousands of tiny files, and the job that reads them the next morning is slow. What happened, and what do you change?

The file count is the number of writing tasks multiplied by the partition values each task holds, so a wide shuffle plus a high-cardinality partition column produces a file explosion. Repartition to match the write layout rather than coalescing to one task, and compact behind a table format.

5 minmid, senior, staff

hardDesignScenario

A graph of two billion edges will not fit on one machine and the algorithm needs neighbours. How do you partition it?

Check the premise first, because two billion edges as vertex id pairs is about 32GB and that fits in one large machine. If it genuinely must be distributed, partition edges rather than vertices, since real graphs are power-law and a single high-degree vertex under an edge cut makes its machine the straggler that every synchronous superstep waits for.

6 minsenior, staff

hardScenarioConcept

A Spark job that used to finish in ten minutes now takes two hours. How do you find out why?

A slow Spark job should be diagnosed from the Spark UI stage list, task duration spread, shuffle volume and physical plan. Common causes are data skew, unnecessary shuffles, missed broadcast joins, repeated recomputation and collect() pulling too much data to the driver.

6 minmid, senior, staff