Questions for data engineering, ML engineering, and the LLM-application roles now appearing in mainstream hiring loops.
13 published across 8 topics.
Data and AI engineering fundamentals58 short answers on one page, for revising rather than studying.Idiomatic Python, pandas and vectorisation, memory behaviour, and writing correct transformations.
Push the work into a whole-column operation so the interpreter is entered once instead of once per row, replace chained indexing with a single .loc assignment, and move to DuckDB, Polars or a cluster once the frame plus its intermediate copies stop fitting in memory.
Batch versus streaming, orchestration, idempotent backfills, and lakehouse and medallion layering.
Make the output of a run a pure function of its partition key and write it by overwriting that partition rather than appending, so a re-run replaces instead of duplicating. Then design for late events, a bounded lookback, and additive schema change, and assume at-least-once delivery into an idempotent sink.
Decide it deliberately, because there is no correct default: you can hold the window open longer, accept the record and restate the figure, or drop it and report how much you dropped. What you must not do is let arrival order silently decide, which is what a pipeline keyed on ingestion time does.
Partitioning and shuffles, skew, joins at scale, catalyst and lazy evaluation, and cluster tuning.
Work from the Spark UI stage list, not the code: laziness means the action only marks where the plan was triggered. Look for a stage whose slowest task dwarfs its median, for shuffle volume a broadcast join would remove, and for a collect() dragging the result through the driver.
Star and snowflake schemas, slowly changing dimensions, and dbt-style transformation layering.
The dimension was overwritten, so historical facts joined to it now report the customer's current region rather than the region they were in when they bought. Keeping history means versioning the dimension row and joining facts to the version that was current at the time, not to the customer.
Settle the grain first - what one fact row means. A star still earns its place for conformed dimensions and type-2 history, while a wide table is a serving artefact for one known pattern. A warehouse differs from a lake by having a modelled contract, not by where the bytes live.
Bias and variance, evaluation-metric selection, leakage, imbalanced data, and validation strategy.
Almost certainly not: if fraud is 0.6% of transactions, a model predicting 'not fraud' for everything scores 99.4% and catches nothing. The metric has to follow from which error costs more, which makes it a business decision expressed in numbers rather than a modelling one.
Overfitting shows up as a validation score materially worse than the training score, but only if the split respects time order and group structure. Fix it by improving the data first, then simplifying, regularising and stopping early, and keep a test set you read once.
Experiment tracking, model registries, deployment patterns, drift detection, and retraining triggers.
Check whether training and serving compute the same features from the same definitions, because the usual cause is two implementations that disagree slightly. Nothing raises an error when they diverge - the model receives plausible numbers that mean something different from the ones it learned on.
Features must be computable at decision time, training and serving must share one transformation implementation, the model and dataset version must be pinned together, drift and realised performance monitored separately because labels arrive late, and rollback must move model and features as one unit.
Embeddings and vector search, chunking, retrieval evaluation, prompt design, and cost and latency control.
Almost always in retrieval rather than generation, and most often in chunking: if the answer was split across a boundary, or sits in a chunk whose surrounding context was stripped, no prompt can recover it. Evaluate the retrieval half separately, because it has a yes-or-no answer that needs no model to grade.
Isolate the layer before changing anything: check whether the correct passage was retrieved at all. If it was not, the fault is in the retrieval stack; if it was, the generator is at fault. Then measure both separately with retrieval recall@k and faithfulness scored on a labelled eval set.
Distributions, hypothesis testing, A/B design, and reading a result honestly.
Fix the hypothesis, one primary metric, the randomisation unit and the sample size before launch. Peeking at a fixed-horizon test inflates false positives because each look is another chance to cross the threshold, and an inconclusive result is an interval to interpret, not proof of no effect.