A DAG that finished in an hour now takes nine and blocks everything behind it. How do you find out where the time went?
Sum the task durations and compare that with the wall-clock span, because nine hours of waiting and nine hours of work are different problems. If the sum is still an hour, you are queueing on a bounded worker pool, usually behind sensors holding slots; if a task genuinely grew, its own duration history over the last thirty runs tells you whether it stepped or sloped.
6 min readUpdated 2026-08-15
0
Answer
Short answer
Diagnose a slow DAG by comparing total task runtime with the wall-clock span of the run. If tasks still add up to about an hour, the nine-hour duration is queueing, pool pressure, sensors, retries or scheduling. If task runtime actually grew, inspect per-task history to find the task that stepped or sloped before tuning the compute engine.
Was this answer helpful?
Likely follow-ups
→Task durations sum to 55 minutes across a nine-hour span. What are the three things you check next, in order?
→Input rows grew three times and runtime grew nine. What operation behaves that way, and how would you confirm it?
→Nothing paged, because the DAG did not fail. What alert would have caught this on the first bad run?
→The DAG is one file with 140 tasks and four unrelated outputs. What would you split, and what does splitting cost you?
Nine hours is either nine hours of work or one hour of work spread across nine hours of waiting. Those have nothing in common. Every minute spent optimising the wrong one is wasted, so the first measurement decides the entire investigation.
Take the sum of every task's own duration for the bad run, and compare it with the wall-clock span from first task start to last task end. Suppose 40 tasks total 55 minutes of runtime while the span is nine hours. Then eight hours is queueing. Nothing inside any task is worth tuning. Suppose instead the tasks total eight and a half hours. Now the work genuinely grew, and you go looking for which task grew.
Most orchestrators expose both numbers. Candidates skip the step because the phrase "the DAG is slow" invites them to think about compute. An interviewer is watching for whether you get the queueing-versus-execution split out of the way in the first thirty seconds.
Loading diagram…
The branch on the right is where the two evidence types diverge: a step change has a date and therefore a deploy next to it, and a slope has a growth rate you can compare against the input.
When the answer is queueing
Queueing means slots. Your workers run some fixed number of tasks at once, and something is holding them without doing work.
Sensors are the classic culprit. A task whose job is to wait for a file to appear occupies a worker slot for its entire wait if it is implemented as a loop that sleeps and re-checks. Put numbers on it: 16 worker slots, 12 of them held by sensors waiting on upstream file drops, leaves four slots for the 40 tasks that have real work to do. A DAG that used to run 16 wide now runs four wide. Nothing in any task got slower. The fix is that a waiting task should not hold a slot - either reschedule mode, where the task exits and is re-queued at the next poke, or a deferrable implementation that hands the wait to a separate process designed for it.
Then check the caps themselves, of which there are several and they compose multiplicatively. A global worker count, a per-pool slot count, a per-DAG concurrency limit, and a per-task concurrency limit are four independent ceilings, and the effective parallelism is the smallest. Someone lowered a pool from 12 to 4 during an unrelated incident and never restored it. That is an ordinary root cause with a nine-hour symptom, and it takes one query to rule out.
The third source is other work. A pool shared with a backfill is not your pool. If somebody kicked off a 90-day reprocessing run, it is consuming the same slots, and your DAG is not slow so much as outbid.
When the answer is a task that grew
If the durations genuinely grew, resist looking at the DAG and look at one task's history. Plot that task's duration for the last thirty runs. That line is the diagnosis.
A step change means a discrete event: a deployment, a configuration change, a dependency upgrade, an index that was dropped, a source table that was repartitioned, a query plan that flipped. The line has a date, and the date has a change log entry beside it. This is by far the more common case and by far the easier to fix, which is why it is worth checking before any theory about data growth.
A gradual slope means the input is growing and the operation does not scale the way you assumed. Do the arithmetic before concluding anything. If input rows grew three times and the task grew nine, that is superlinear, and linear or log-linear work does not behave that way - three times the rows through a sort would be roughly three and a bit. Nine from three points at something quadratic: a join that lost its predicate and became a cross product for some keys, a per-row lookup executed inside a loop, or a full scan where an index used to be selected.
The subtle one is retries. A task configured with three retries and a 30-minute timeout that fails twice before succeeding contributes 90 minutes and reports success. The DAG is green, the graph shows one duration, and the run is three times longer than the work in it. Check the try number on the slow tasks, because a pipeline quietly retrying its way to correctness looks healthy in every view except the wall clock.
Why everything behind it is blocked, which is a design finding
The second half of the question is not incidental. That downstream pipelines are blocked tells you something about the topology, and it is usually one of three things.
They share a pool with this DAG, so the nine-hour run is starving them of slots directly. They depend on this DAG's completion through a sensor or a dataset trigger, so they are correctly waiting for data that has not arrived. Or the whole thing is one enormous DAG producing four unrelated outputs, so a slow branch delays outputs that had no reason to depend on it.
The first two are configuration and the third is architecture. Splitting a monolithic DAG along output boundaries means an unrelated output is no longer hostage to this one, at the cost of making the cross-DAG dependencies explicit and therefore visible - which is usually an improvement, because implicit dependencies inside one file are the ones nobody can enumerate.
There is a monitoring finding here too, and it is the one worth volunteering. Nothing paged. The DAG did not fail. It succeeded, eight hours late. If your alerting only fires on failure, then every latency regression in every pipeline is invisible until a human notices a stale dashboard. What catches this on the first bad run is an alert on the span itself: this DAG must complete by 07:00, and if it has not, page. Expressing that as a deadline rather than as a duration also survives the input growing, because what consumers care about is when the data is ready and not how long it took.
The mistake that costs a day
Reaching for the compute engine first. Someone says nine hours, and the reflex is to open the Spark UI, look for skew, and start tuning partitions. If eight of those nine hours were spent waiting for a slot, every one of those changes is a no-op measured against the symptom, and you will not know why your fix did nothing.
The ordering is the skill: span against sum, then slots or task, then step or slope, and only then engine internals. Each step costs a minute and eliminates a whole class of cause. State it in that order in an interview even if you happen to guess the answer, because the ordering is what an interviewer can grade.
Nine hours of span is not nine hours of work until you have checked. Compare the sum of task durations against the wall clock first, and let that one number decide whether you are looking at a scheduler or at a query.