Data and AI
The work of moving data reliably, modelling it so people can trust an answer, and putting statistical or language models into production. Five different jobs hide under one label, and confusing them is the most common reason a career move here goes sideways.
Assumes you know: Python at a working level, including reading someone else's code, SQL beyond joins - grouping, window functions, and reading a query plan, Enough statistics to know what an average hides, For the engineering roles - version control, testing, and how a deployment works
Overview
What this area actually covers
Four things, plus one recent arrival. Getting data from where it is produced to where it can be queried, which is data engineering. Shaping that raw data into models a business can reason about, which is analytics engineering. Using it to answer questions with statistics, which is data science. Running learned models in production so they keep working, which is machine learning engineering and MLOps. And building applications on top of large language models, which is new enough that its job title is still unsettled.
These are genuinely different jobs. A data engineer's week is pipeline correctness, schema evolution and a 3 a.m. page about a failed load. A data scientist's week is framing a question, arguing about whether the measurement is valid, and communicating an uncertain answer. They both write Python and the resemblance ends there.
The boundary people get wrong is with analytics and reporting. Building a dashboard is not data engineering, and someone whose job is dashboards will not enjoy being handed a Spark job. The other misplacement is research: designing new architectures and publishing on them is a small, credential-heavy field, and almost nothing advertised as an AI job is that.
What gets wrongly bundled in: business intelligence tooling, which is its own craft; data governance, a policy discipline that consumes engineering effort but is not engineering; and prompt writing, which is a skill inside LLM application work rather than a role.
Where to go next
Now practise it
40 interview questions in Data & AI Engineering, each with the rubric the interviewer is scoring against.
- Events for a daily revenue table arrive up to three days late. How do you design the pipeline and the backfill?
- Your test lifts conversion, but something else moved the wrong way. What are guardrail metrics, and when do they stop a launch?
- You must ingest all of English Wikipedia every day, and the source does not want to be hammered. What runs, how often, and what do you do about the run that fails at hour six?
- A transform has been writing wrong revenue figures for three days and six downstream tables have consumed it. How do you backfill the corrected data without double-counting anything?