Data and AI

What this area actually covers

Four things, plus one recent arrival. Getting data from where it is produced to where it can be queried, which is data engineering. Shaping that raw data into models a business can reason about, which is analytics engineering. Using it to answer questions with statistics, which is data science. Running learned models in production so they keep working, which is machine learning engineering and MLOps. And building applications on top of large language models, which is new enough that its job title is still unsettled.

These are genuinely different jobs. A data engineer's week is pipeline correctness, schema evolution and a 3 a.m. page about a failed load. A data scientist's week is framing a question, arguing about whether the measurement is valid, and communicating an uncertain answer. They both write Python and the resemblance ends there.

The boundary people get wrong is with analytics and reporting. Building a dashboard is not data engineering, and someone whose job is dashboards will not enjoy being handed a Spark job. The other misplacement is research: designing new architectures and publishing on them is a small, credential-heavy field, and almost nothing advertised as an AI job is that.

What gets wrongly bundled in: business intelligence tooling, which is its own craft; data governance, a policy discipline that consumes engineering effort but is not engineering; and prompt writing, which is a skill inside LLM application work rather than a role.

The eight topics underneath

The section splits into eight topics, and they are best read as three groups. The first four follow data from a Python transformation through pipelines and distributed compute into a modelled warehouse. The next three follow a model from training through production and into the language-model applications built on the same platform. The last one, statistics, underpins the honesty of everything above it.

Python for Data is the working language of the whole section, and the interview content is narrower and more specific than general Python. Idiomatic use of the standard library, pandas and the difference between a vectorised operation and a loop that happens to be written in a dataframe, how memory behaves when you copy a frame you thought you were mutating in place, and how to write a transformation that is deterministic and testable. It exists as its own topic because most data defects are introduced here, in code that runs without error and produces something subtly wrong.

Data Pipelines covers the movement itself: batch against streaming and the honest reasons to choose either, orchestration and what a scheduler is really giving you, idempotency so that a rerun does not double-count, backfills that can be replayed without corrupting what is already there, and the lakehouse layering conventions in which raw, cleaned and modelled data live in distinct zones. This is the topic that most distinguishes data engineering from application engineering, because everything in it is about what happens when a run fails halfway.

Spark & Big Data is the distributed-computation topic, and its questions are almost all about the same underlying thing: where data has to move between machines. Partitioning, the shuffle and what triggers one, skew where one partition holds most of the rows and one task holds up the whole job, join strategies and when broadcasting the small side is the right call, lazy evaluation and how a query plan gets optimised before anything executes. It is a separate topic because the mental model — think about data movement, not about lines of code — is not one that transfers from single-machine work.

Warehousing & Modelling is where raw data becomes something a business can reason about. Dimensional modelling with facts and dimensions, star and snowflake shapes and what each costs at query time, slowly changing dimensions and the several strategies for retaining history, grain and why stating it precisely prevents most modelling arguments, and the layered transformation approach that dbt made conventional. The ideas here are decades old, which is a recommendation rather than a criticism.

ML Fundamentals is the topic that determines whether a model result means anything. Bias and variance and how to recognise which one is hurting you, choosing an evaluation metric that matches the decision the model informs, leakage in its several forms, handling class imbalance without fooling yourself, and validation strategy — which is the single highest-value idea in the topic, because a wrong split produces an optimistic number and no error message. Notably, none of this is about algorithms; the algorithms are in libraries and the judgement is not.

MLOps is what happens after the model works once. Experiment tracking so a result can be reproduced, a registry so you know which version is serving, deployment patterns including shadow and staged rollouts, monitoring for drift in the input distribution and in the relationship being modelled, and the triggers that decide when to retrain. It is its own topic because the failure mode it addresses is silent: a model does not crash when it becomes wrong, it simply keeps returning confident predictions.

LLMs & RAG covers applications built on language models, which in practice means retrieval far more than it means models. Embeddings and how vector search differs from keyword search, chunking and why the boundaries you choose determine what can ever be retrieved, evaluating retrieval separately from generation, prompt design as an engineering activity with versions and tests, and controlling cost and latency. It is the fastest-moving topic here and the one where the durable skills are least like the tooling.

Statistics & Experimentation sits last and underwrites everything. Distributions and what they imply about the summary statistics you quote, hypothesis testing and what a p-value does and does not say, designing an A/B test including how long it must run and what you must decide before it starts, and reading a result honestly when it is inconclusive or when someone has looked at it thirty times. It is separated out because it is the topic that most often distinguishes a data scientist from an analyst in an interview.

Where it sits in a real system

Data has a direction of travel. Applications and third parties produce events and records. Ingestion lands them somewhere durable and cheap, historically a lake of files, now usually a table format over object storage that supports transactions. Transformation turns that landing zone into a modelled layer, conventionally staging, then core facts and dimensions, then marts shaped for particular consumers. Everything downstream - dashboards, features for models, exports to a finance system, a retrieval index - reads the modelled layer, not the raw one.

flowchart TD
    A[Applications and third parties produce records] --> B[Ingestion lands raw data durably]
    B --> C[Staging cleans types and deduplicates]
    C --> D[Core facts and dimensions, one grain each]
    D --> E[Marts shaped for a specific consumer]
    E --> F[Dashboards, model features, exports, retrieval index]
    B -.-> F

The dotted arrow is the one that matters, because it is the shortcut every organisation takes under deadline pressure: a consumer reaching past the modelled layer directly into raw data. It works immediately and it is what makes every subsequent upstream schema change an incident, because a change that the modelled layer would have absorbed now breaks something nobody knew was reading it.

That last point is where the discipline lives. When consumers reach past the modelled layer into raw tables, every upstream schema change becomes an incident, and the same business metric ends up computed three different ways with three different values. The reason the argument about "one source of truth" recurs in every company is that it is a real structural property, not a slogan.

Machine learning attaches to this pipeline twice, and the two attachments must agree. Training reads history to fit a model. Serving reads current values to make a prediction. If the two compute a feature differently - training averages a customer's last thirty days including today, serving excludes today - the model degrades in production for reasons no metric on the training set will reveal. That mismatch, training/serving skew, is the most common serious ML production bug and it is a data-plumbing failure rather than a modelling one.

LLM applications hang off the same data platform. Retrieval-augmented generation means fetching relevant text at request time and putting it in the prompt so the model answers from your content rather than its parameters. The retrieval half is a data pipeline: extract documents, split them sensibly, embed them, index them, keep the index current as sources change, and handle deletion when a document should no longer be reachable.

Batch, streaming, and the argument between them

The most common design question in this section is whether a pipeline should run on a schedule or continuously, and it is asked because the wrong answer is expensive in both directions. It is worth being able to reason about rather than having a preference.

Batch processing runs over a bounded set of data at a known time. Its great virtue is that it is easy to reason about and easy to fix: the input is fixed, so a run is reproducible, and when something is wrong you rerun it. Its cost is latency, and the latency is not the schedule interval but roughly the interval plus the run time, which people consistently underestimate when promising a business that data will be "hourly".

Streaming processes records as they arrive, and the reason it is harder is not the infrastructure but time itself. In a stream you have two clocks — when the event happened and when you received it — and they disagree, because a phone was offline or a queue backed up. That forces decisions a batch job never has to make: how long to hold a window open waiting for stragglers, what to do with a record that arrives after you have already emitted a result for its window, and whether your output is allowed to be corrected later. Exactly-once processing, which is the guarantee everyone asks for, usually turns out on inspection to be at-least-once delivery plus idempotent writes, which is a meaningfully different and much more achievable thing.

The decision should follow from what the data is for, and the honest question is what someone will do differently with fresher data. If a dashboard is read at nine each morning, hourly loading is spending money for nothing. If a fraud decision has to be made before a payment completes, no batch schedule is fast enough. Most requirements sit in between, and the mature answer is often a batch pipeline for the modelled layer with one streaming path for the specific decision that needs it, rather than converting everything because one consumer needed low latency.

The consequence for interviews is that "we would use streaming" is not an answer on its own. What is graded is whether you named the latency requirement first, whether you know what late data does to your results, and whether you can say how the pipeline recovers after a failure — because in streaming, recovery means replaying from a retained offset rather than rerunning yesterday, and the retention window is now a design parameter someone has to choose.

Telling the five jobs apart

Because the label covers so much, the most useful thing this page can give you is a way to work out which job an advert is describing and which one you want. The reliable test is not the technology list, which is nearly identical across all five, but the artefact the role is judged on.

The column that predicts job satisfaction best is the last one, and it is the one candidates never ask about. Being on call is a proxy for whether your output is a running system or a document, and people who want one and are given the other are the ones who leave within a year. The second most predictive column is "judged on": a data scientist whose work does not change any decision has had an unsuccessful year regardless of how good the models were, which is a very different accountability from a pipeline either running or not.

There is one more reading of the table worth making. Every row shares the same prerequisites — Python, SQL, and enough statistics not to mislead yourself — which is why moving between rows is possible and why the adverts blur. What differs is which failure you are accountable for, and that is worth deciding deliberately rather than discovering after you accept.

A useful way to hear a job advert: if it says Python and machine learning but the responsibilities are all pipelines and orchestration, it is a data engineering role. If it says data engineering but the responsibilities are dashboards, it is analytics. Both happen constantly.

Who does this work

The builders are data engineers owning ingestion and the platform, analytics engineers owning the transformation layer, machine learning engineers taking models to production, and platform or MLOps engineers running the training and serving infrastructure. Increasingly there are AI engineers building LLM-backed features, who are usually application engineers who learned retrieval and evaluation rather than researchers who learned web development.

The specifiers and consumers are data analysts, product managers who own a metric, and data scientists on the experimentation side. A large amount of a data scientist's real day is neither modelling nor coding: it is establishing what a stakeholder actually wants to know, then finding out whether the data can honestly answer it. Beside them sit analytics leads and governance functions who decide what a metric means and who may see it.

The size of a company changes these boundaries more than anything else does. In a small one, a single person does all five jobs and the title is whichever one the founder wrote down; in a large one, the roles are distinct enough that moving between them is a career change with an interview attached. That matters when you choose where to start, because a small company teaches you the whole pipeline shallowly and a large one teaches you one segment of it properly, and the two produce very different candidates three years later.

Demand, adoption and how that is changing

Demand is very high, and the reasons differ per sub-role. Data engineering demand is structural: the platform migration from on-premise warehouses to cloud object storage plus a separate compute engine is still under way at a great many organisations, and every one of those is multi-year work with a long maintenance tail afterwards. Cost pressure adds to it rather than subtracting, because separating storage from compute made spend visible and someone now has to own it.

LLM application demand is the fastest-moving and the least stable. Organisations that spent 2023 and 2024 on demonstrations are now trying to run these systems as products, which surfaces unglamorous requirements: evaluation, latency budgets, cost per request, handling the cases where the model is confidently wrong, and access control on retrieved content. That shift is good news for engineers and bad news for demonstration-builders, because the durable skills turn out to be retrieval quality, evaluation design and cost control rather than anything about training models.

Be honest about what is commoditising. Training a general-purpose model from scratch is out of reach for almost everyone and getting further out of reach. Fine-tuning is narrower in application than it was assumed to be, because retrieval and prompt design solve a large share of the cases people reached for fine-tuning to solve. Classical modelling has largely been absorbed into libraries: writing gradient descent by hand is a teaching exercise, and the value moved to feature construction, validation design and deployment. Meanwhile the boring end - orchestration, data quality, lineage, warehouse cost - is not commoditising at all, which is exactly why it stays employable.

The access-control point deserves its own sentence, because it is where a great many internal LLM projects have stalled. A retrieval index built over a company's documents inherits none of the permissions those documents had, so a naive implementation will happily quote a document to someone who could not have opened it. Solving that means filtering at retrieval time against the requesting user's entitlements, which is an engineering problem with no clever prompt available as a substitute, and it is the kind of requirement that separates a demonstration from a product.

What makes it hard

Most machine learning projects do not fail on the model. They fail because the problem was framed wrongly or the data cannot support the claim. Framing failure looks like this: a business wants to "reduce churn", so someone trains a churn classifier, and it turns out the label was defined as cancelled within thirty days, which mostly identifies customers who already decided, leaving no time to act. The model can be accurate and useless simultaneously. Nothing in the training metrics detects this, because the metric measures agreement with a label nobody interrogated.

Data quality failures are quieter and more expensive. Leakage is the canonical one: a feature that would not be available at prediction time leaks the answer into training, so validation scores look excellent and production performance collapses.

-- Leaky: refund_amount is only populated after the order is resolved,
-- so a model trained on it "predicts" fraud it is actually being told about.
select order_id, customer_age_days, basket_value, refund_amount, is_fraud
from orders;

-- Honest: only fields whose values exist at the moment of scoring.
select o.order_id,
       o.customer_age_days,
       o.basket_value,
       count(p.id) as prior_orders   -- prior, strictly before this order
from orders o
left join orders p
  on p.customer_id = o.customer_id
 and p.created_at  < o.created_at
group by 1, 2, 3;

The second query is harder to write, slower, and correct. That gap - between the easy query and the temporally honest one - is a large part of what applied ML skill consists of.

The genuinely non-substitutable experience is in validation design. Random train/test splitting is wrong whenever the data has time structure or grouping, and it fails silently rather than loudly: you get an optimistic number and no warning. Knowing to split by time for a forecasting problem, by entity when the same customer appears in many rows, and to check whether your split leaks a group across the boundary is learned by having been burned.

Choosing the metric is the same kind of trap in a different place. Accuracy on an imbalanced problem is meaningless — a fraud model that predicts "not fraud" for everything can be right almost all of the time and catch nothing — and the metric you should use depends on which error costs more. Precision matters when acting on a positive is expensive, recall matters when missing one is, and the threshold between them is a business decision rather than a modelling one. Candidates who can connect the metric to the cost of the two error types are demonstrating the thing the whole topic is for.

Data engineering is hard in a different register, and the honest description is that it is software engineering with the reliability and correctness dials turned up. The code is not intellectually exotic. What is hard is that pipelines run unattended, on data whose shape you do not control, and a defect corrupts a state that persists - so recovery means backfilling, not restarting. That forces habits many application engineers have never needed: idempotent tasks so a retry does not double-count, explicit handling of late-arriving records, awareness that a partial write leaves a partially correct table, and treating schema change from an upstream team as an expected event rather than an outage. Anyone who tells you data engineering is not real engineering has not maintained a pipeline whose failure was noticed a week later by finance.

LLM work has its own difficulty, and it is evaluation. Traditional software is testable against expected output; a generative system has many acceptable outputs and its failures are plausible-sounding. Without an evaluation set - real inputs, with judgements about what a good response looks like - you cannot tell whether a prompt change helped, and every iteration becomes vibes. Building that set is unglamorous, is where most of the leverage is, and is the thing teams skip. Retrieval quality has the same character: if the right passage is not in the context, no amount of prompt tuning recovers it, so the failure usually lives in chunking and indexing rather than in the model.

Following one retrieval request

Because retrieval is where most LLM application failures occur, it is worth seeing the request path once. Almost every question about latency, cost, permissions or wrong answers resolves to one of these steps.

sequenceDiagram
    participant U as User
    participant A as Application
    participant V as Vector index
    participant M as Language model
    U->>A: Asks a question
    A->>A: Embeds the question into a vector
    A->>V: Retrieves the nearest chunks
    V-->>A: Returns candidates with scores
    A->>A: Filters by permission and reranks
    A->>M: Sends the prompt with the chosen chunks
    M-->>A: Returns an answer
    A-->>U: Answer plus citations to the chunks used

The step to examine is the filtering one, because it is the step most implementations omit and the one that carries both the permission problem and most of the quality problem. Nearest-neighbour search returns what is closest in embedding space, which is not the same as what is most relevant, and a reranking pass over a slightly larger candidate set is usually the cheapest available quality improvement.

The diagnostic that follows from the diagram is that you can and should evaluate the retrieval half on its own. Given a question, was the passage containing the answer among the retrieved chunks? That question has a yes or no answer, needs no model to grade, and tells you immediately whether a disappointing response is a retrieval failure or a generation one. Teams that skip it end up tuning prompts against a problem the prompt cannot reach.

Why study it

Study data engineering if you like systems where correctness is the product and you are comfortable being paged. The skills transfer widely, the demand is not fashion-driven, and the knowledge ages slowly: dimensional modelling ideas from decades ago still describe how a warehouse should be shaped.

Study LLM application work if you want the fastest-moving area with the most hiring, and accept that specific tooling knowledge will depreciate quickly while the underlying skills - retrieval, evaluation, cost and latency management, failure handling - will not.

Study the statistics if you intend to be the person in the room who can say whether a result is real. That capability is rarer than it should be and it is the thing that makes an analyst's opinion carry weight, because most disputes about data are not about the number but about whether the comparison behind it was fair.

Do not bother if you want to do research: that path runs through a postgraduate degree and publications, not through a bootcamp, and almost no advertised job is it. Do not enter data science expecting mostly modelling, because most of the job is stakeholder work and data archaeology, and people who dislike that leave. And if your real goal is simply a well-paid engineering job, backend engineering is a shorter and more reliable route than any of these.

Your first hour

Take a dataset with time in it - your bank statements exported to CSV, or a public dataset with dates - and predict something one step ahead. Build it twice. First, split rows randomly into train and test and record the score. Then split by time, train on everything before a cut-off, test on everything after, and record the score again. Write down both numbers and one sentence explaining the difference.

The artefact is those two numbers plus that sentence. It is a small thing that demonstrates you understand why the first number was a lie, which is a more convincing interview answer than a list of algorithms you have used.

For the retrieval side, spend the hour differently: take twenty pages of documentation you know well, write ten questions you know the answers to, then check by hand whether a naive chunking of that text would even contain each answer in one chunk. You will find two or three that split across a boundary. That is the whole problem of RAG, discovered in an hour without writing a line of code.

For the pipeline side there is a third version of the exercise, and it is the one that teaches the habit data engineering is built on. Write a small script that loads a CSV into a table, then run it twice and count the rows. If the count doubled, you have written the non-idempotent version that every beginner writes, and the fix — a natural key with an upsert, or deleting the partition you are about to write before writing it — is the single most important pattern in the whole topic. Ten minutes and a duplicated row teach it better than any amount of reading.

What this is not

It is not one job. "Data and AI" on a CV means nothing without which of the five roles you are describing, and hiring managers read the vagueness as inexperience.

It is not mostly modelling. Across every role here, the majority of time goes on data acquisition, correctness, plumbing and communication. Anyone whose interest ends where the model fitting ends will be unhappy.

It is not machine learning research, and the two are frequently conflated by people entering the field. Applied work is about deploying and validating known methods against messy data; research is about creating methods, in a much smaller job market with much stricter entry requirements.

It is not a field where the tooling is the skill, however the adverts read. The list of named technologies changes every few years while the underlying questions — what is the grain of this table, is this comparison fair, what happens when this rerun overlaps the previous one, would this feature have been available at prediction time — do not. A candidate who can answer those in the abstract learns any given tool in a week; the reverse is not true.

Nor is it a field where a dashboard is the deliverable. A chart nobody acts on has the same value as no chart, and the roles here are ultimately judged on whether a decision, a product behaviour or a running system changed. That is why communication appears in every honest description of a data scientist's job and why data engineers are measured on whether their pipeline runs rather than on how it was written.

It is not made obsolete by better models. Improved models raise the value of the data feeding them and the evaluation judging them, which is where the labour actually is.

Almost every failure you will see in this area is a framing or data failure wearing a modelling failure's clothes, which is why the first question to ask of any model is not how accurate it is but whether the label means what someone assumed.