hardScenarioConceptMidSeniorStaff
Your nightly Spark job writes tens of thousands of tiny files, and the job that reads them the next morning is slow. What happened, and what do you change?
5 min readUpdated 2026-08-15Target archetype: Big Tech, Enterprise Captive, Product Startup
Answer
Short answer
The file count is the number of writing tasks multiplied by the partition values each task holds, so a wide shuffle plus a high-cardinality partition column produces a file explosion.
Was this answer helpful?
Likely follow-ups
- You cannot change the partition column because downstream queries filter on it. What is left to you?
- Why does coalesce to one file sometimes make the whole job slower rather than just the write stage?
- How would you decide the target file size, and what does going too large cost you?
- What does a table format give you here that a folder of Parquet files does not?
Related questions
- A Spark job that used to finish in ten minutes now takes two hours. How do you find out why?hardAlso on spark and shuffle6 min
- A transform has been writing wrong revenue figures for three days and six downstream tables have consumed it. How do you backfill the corrected data without double-counting anything?hardAlso on partitioning4 min
- Consumer lag on your orders topic climbs every morning and never clears before the next peak. Work through it.hardAlso on partitioning6 min
- You read a four gigabyte CSV into pandas and the process needs far more memory than that. Where did it go?mediumAlso on parquet4 min