hardScenarioConceptMidSeniorStaff
A Spark job that used to finish in ten minutes now takes two hours. How do you find out why?
A slow Spark job should be diagnosed from the Spark UI stage list, task duration spread, shuffle volume and physical plan. Common causes are data skew, unnecessary shuffles, missed broadcast joins, repeated recomputation and collect() pulling too much data to the driver.
6 min readUpdated 2026-08-15
Answer
Short answer
Start in the Spark UI, not in the code. Find the slow stage, compare max task time to median task time, inspect shuffle read/write and confirm the physical plan; skew, shuffle, recomputation and driver collect() are the usual culprits.
Was this answer helpful?
Likely follow-ups
- You confirm one key holds 60% of the rows. What are your options, and what does each cost?
- When is repartition the wrong tool and coalesce the right one, and when is that reversed?
- Adaptive query execution is already on. What classes of problem does it still not solve?
- The same job is fast on the sample and slow in production with identical code. Where do you look?
Related questions
- Your nightly Spark job writes tens of thousands of tiny files, and the job that reads them the next morning is slow. What happened, and what do you change?hardAlso on spark and shuffle5 min
- Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?hardSame kind of round: concept5 min
- A method with REQUIRES_NEW was meant to save an audit row even when the main transaction fails. When does it work, and when does it deadlock?hardSame kind of round: concept5 min
- What does a forward deployed engineer interview test that a standard software engineer loop does not?mediumSame kind of round: concept5 min