Batch versus streaming, orchestration, idempotent backfills, and lakehouse and medallion layering.
11 questions in Data & AI Engineering.
Land raw events unchanged by arrival date, then rebuild each revenue day from that raw layer by event date. Each run overwrites only the days its new arrivals touched, a day is provisional until the three-day window closes, and the backfill is the same job run over a date range, never a separate script.
Do not re-download Wikipedia daily. Use incremental ingestion: load the published dump once, follow the recent-changes feed, and fetch only the latest revision of each changed page in small, polite batches. Checkpointing each batch behind a revision-guarded write lets a run that fails at hour six resume.
Make the write idempotent before you make it correct: partition-level replace rather than append, so rerunning a day produces the same result whether it runs once or five times. Then work out how far the contamination spread, because a downstream table that accumulated rather than recomputed will not fix itself when the source is repaired.
Sum the task durations and compare that with the wall-clock span, because nine hours of waiting and nine hours of work are different problems. If the sum is still an hour, you are queueing on a bounded worker pool, usually behind sensors holding slots; if a task genuinely grew, its own duration history over the last thirty runs tells you whether it stepped or sloped.
Validate every record against a declared schema at the ingestion boundary, quarantine what fails instead of dropping it, treat additive changes as safe and renames as breaking, and alert on a field that has gone all-null rather than waiting for a dashboard to look wrong.
An authoritative discussion on Apache Iceberg's table format, focusing on hidden partitioning, the trade-offs of copy-on-write versus merge-on-read, and managing schema drift at scale. Use this data engineering answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects data lakes to the point an interviewer is testing.
An examination of decentralised data governance, exploring dynamic masking in the compute layer, deterministic tokenisation for referential integrity, and standardised tagging. Use this data engineering answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects data governance to the point an interviewer is testing.
Design a safely rerunnable data pipeline by making each output partition a deterministic result of its input partition, overwriting instead of appending, and writing into idempotent sinks that tolerate retries and late events.
Decide it deliberately, because there is no correct default: you can hold the window open longer, accept the record and restate the figure, or drop it and report how much you dropped. What you must not do is let arrival order silently decide, which is what a pipeline keyed on ingestion time does.
An analysis of distributed processing bottlenecks, addressing the limitations of AQE, implementing manual key salting for shuffle optimization, and restructuring data layouts. Use this data engineering answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects batch processing to the point an interviewer is testing.
Designing a robust streaming architecture for embedding generation, detailing GPU adaptive batching, Kafka-driven backpressure management, and asynchronous HNSW indexing. Use this machine learning answer to show the decision, trade-off, and evidence rather than a memorised definition. It also connects data engineering to the point an interviewer is testing.