Loading...
Loading...
Browse 4 real-world technical and behavioral interview questions about Data pipelines. Review scenarios, edge cases, and architectural best practices.
Make the write idempotent before you make it correct: partition-level replace rather than append, so rerunning a day produces the same result whether it runs once or five times. Then work out how far the contamination spread, because a downstream table that accumulated rather than recomputed will not fix itself when the source is repaired.
Sum the task durations and compare that with the wall-clock span, because nine hours of waiting and nine hours of work are different problems. If the sum is still an hour, you are queueing on a bounded worker pool, usually behind sensors holding slots; if a task genuinely grew, its own duration history over the last thirty runs tells you whether it stepped or sloped.
Validate every record against a declared schema at the ingestion boundary, quarantine what fails instead of dropping it, treat additive changes as safe and renames as breaking, and alert on a field that has gone all-null rather than waiting for a dashboard to look wrong.
Decide it deliberately, because there is no correct default: you can hold the window open longer, accept the record and restate the figure, or drop it and report how much you dropped. What you must not do is let arrival order silently decide, which is what a pipeline keyed on ingestion time does.