The distance between a model that works once and a model a business depends on. Reproducibility, point-in-time correct training data, drift you must detect without labels, GPU economics and model governance are the work, and none of it is modelling.
High demand21 min readMLOps Engineer, ML Platform Engineer, Machine Learning Engineer, Data / Platform Engineer moving into ML, Site Reliability Engineer for ML systems, Model Risk and Validation AnalystUpdated 2026-07-28
Assumes you know: Comfortable with containers, CI/CD and infrastructure as code, Understanding of how a model is trained and evaluated, even if you do not train them, Some experience operating a production service and being accountable for it
Overview
What this area actually covers
This section is about the gap between a model that produced a good number in a notebook and a
model a business depends on. Closing that gap means being able to reproduce the result, having
training data that reflects only what was knowable at the time, running training as a scheduled
pipeline rather than a script somebody remembers how to invoke, promoting a version through
gates that can reject it, serving predictions inside a latency budget on hardware you can
afford, detecting that the world has moved before your users do, and being able to answer an
auditor's question about a decision made eighteen months ago.
None of that is modelling. That is the defining characteristic of the discipline and the reason
it is a separate section: the skills are software engineering, infrastructure, data engineering
and operations, applied to an artefact with two properties ordinary software does not have. The
first is that its behaviour is determined by data as much as by code, so versioning the code
version is not enough to reproduce anything. The second is that it degrades without changing —
the code is identical, the deployment is untouched, and the model gets worse because the world
it was fitted to has moved. Every unusual practice in this section descends from one of those
two facts.
The boundary with adjacent work is worth drawing precisely, because the roles are hired
separately. Choosing a model family, engineering features for predictive power, selecting a
metric and reasoning about leakage is Machine Learning. Building the
pipelines that populate a warehouse or lake is data engineering, in
Data & AI Engineering. Serving and evaluating language models has its own
operational discipline covered as LLMOps within
Generative AI & Large Language Models — the concepts here transfer, but
the artefacts differ, because there you version prompts and configuration against somebody
else's weights rather than versioning your own.
On this site's structure: the older Data & AI Engineering section still
carries an MLOps subsection from an earlier authoring wave, and it has not been removed. Treat
it as the compressed introduction to what this section covers across eight subsections.
Two things are wrongly bundled in. The first is "MLOps is DevOps for models", which is true
enough to be misleading: the pipeline, container and deployment craft does transfer directly, and
the parts that do not transfer — point-in-time correct training data, evaluation gates, drift
detection without labels, and the fact that a rollback restores code but not necessarily
behaviour — are exactly the parts that make the job difficult. The second is tool knowledge as a
substitute for the discipline. Naming an experiment tracker, an orchestrator and a registry
describes a stack; explaining what has to be captured for a result to be reproducible describes
the understanding those tools exist to encode.
The order follows a model's life. The first two are what you need before training is meaningful,
the next two are how a model gets built and promoted, then how it serves, then how you find out
it has stopped working, and the last two are the constraints that decide whether the whole
arrangement is sustainable — money and permission. Reproducibility is first because everything
else is unverifiable without it.
Subsection
What it is for
Experiment Tracking & Reproducibility
Being able to recreate a result and prove which change caused it
Feature Stores & Training Data
Consistent features offline and online, correct as of a point in time
Training Pipelines & Orchestration
Turning a script into a scheduled, recoverable, validated process
Model Registry & CI/CD
Promoting and rolling back a model the way you would code
Serving & Inference Infrastructure
Delivering predictions inside a latency budget and a hardware budget
Monitoring, Drift & Retraining
Noticing degradation before users do, usually without labels
Cost & Capacity Management
Knowing the unit cost of a prediction and controlling it
Governance & Model Risk
Documentation, fairness, explainability and audit where they are required
Experiment Tracking & Reproducibility
What must be captured for a result to be recreatable: the code version, the data version, the
configuration, the environment, the random seeds, and the hardware where it affects numerics. It
covers data and code versioning together, sources of nondeterminism that survive a fixed seed,
and the reason a notebook is a workspace rather than a record. It is first because it is the
foundation of every claim anyone makes in this field: without it, "the new model is better" is
an assertion about two runs that differed in ways nobody recorded. Interviewers ask what you
would need to reproduce a six-month-old result, and the strong answer includes the data snapshot
and the environment, not just a commit hash.
Feature Stores & Training Data
Parity between the features computed for training and those computed at serving time,
point-in-time correctness so a training row contains only what was knowable at its timestamp,
backfilling a new feature over history, collecting labels and coping with the delay before they
arrive, and the honest question of whether a feature store is worth the operational cost for a
given team. It is separate because this is where the hardest correctness problem in the section
lives. Train-serve skew produces a model that validated well and underperforms in production for
reasons no monitoring will explain, and the cause is almost always that two pieces of code
computed the same feature differently.
Training Pipelines & Orchestration
Converting a training script into a scheduled pipeline with dependencies, data validation gates
that stop a run when the input is wrong rather than training on it, distributed training when a
single machine is insufficient, checkpointing, and recovering from a failure partway through a
long job. Its own subsection because the transition from script to pipeline is where most of the
engineering rigour enters, and because the validation gate is the highest-value component: a
pipeline that trains happily on a day when an upstream job silently produced nulls will deploy a
worse model on schedule and tell nobody.
Model Registry & CI/CD
Promotion from development to staging to production, champion and challenger arrangements,
automated gates that compare a candidate against the incumbent on a fixed evaluation set,
rolling a model back, and treating a model version as a deployable artefact with a change
history. Separate because the rules differ from code deployment in a way that matters: a code
rollback restores previous behaviour deterministically, whereas rolling back a model returns you
to a version fitted to older data, which may itself now be wrong. Expect material on what a
promotion gate should actually check, which is more than a single metric threshold.
Serving & Inference Infrastructure
Batch, online and streaming inference and how to choose; latency budgets and where the model
sits within one; autoscaling and the cold-start problem when a container must load large
weights; sharing GPUs between workloads; and shadow deployments that run a new model on live
traffic without acting on its output. It exists as its own area because serving is where
ordinary distributed systems engineering meets the specific awkwardness of model artefacts —
large, slow to load, occasionally requiring specialised hardware — and because the choice of
serving mode is frequently the decision that makes a project affordable or not.
Monitoring, Drift & Retraining
Distinguishing data drift, where the inputs have changed distribution, from concept drift, where
the relationship between inputs and outcome has changed; monitoring quality when labels arrive
weeks late or never; choosing a signal worth alerting on rather than one that fires constantly;
and deciding whether a retrain is warranted or whether the problem is upstream. This is the
subsection that has no analogue in conventional operations, and it contains the field's hardest
open question: how to tell that a model has degraded when you cannot yet observe whether it was
right. Proxy signals, prediction distribution shifts, and deliberate holdouts are the tools, and
each has limits worth knowing precisely.
Cost & Capacity Management
The economics of accelerated hardware, using interruptible capacity for work that tolerates it,
right-sizing inference rather than provisioning for a peak that occurs weekly, caching
predictions where inputs repeat, and being able to state the unit cost of a single prediction.
It is separate because cost in machine learning systems is unusually concentrated and unusually
visible: a small number of instances can dominate a platform bill, and an ML platform team is
routinely asked to defend that line. Being able to express a model's cost per thousand
predictions, and to say which architectural choice drives it, is a differentiating skill.
Governance & Model Risk
Model documentation, formal approval before deployment, fairness and bias testing, providing
explanations where a regulation or a customer requires them, audit trails that can reconstruct a
past decision, and the model risk management practice established in financial services and now
spreading. It closes the section because in regulated industries it is the gate everything else
must pass. It is also the subsection that most changes what an interview looks like: a bank or an
insurer will ask how a model was validated by someone independent of its builder, and a candidate
who has only worked where deployment was an engineering decision will not have an answer.
Where it sits in a real system
An ML platform sits between the data platform and the product, and it takes a dependency in each
direction that a normal service does not. Upstream it depends not merely on a data system being
available but on its semantics being stable: a column whose meaning changes, a pipeline that
starts emitting nulls, a definition adjusted for a reporting need, will each degrade a model
without producing a single error anywhere. That is why data validation gates sit inside training
pipelines rather than being left to the data team.
Downstream it serves a product decision, which means a prediction has a consumer with an
expectation attached. Latency budgets arrive from that consumer. So does the requirement that the
system degrade usefully: a recommendation service that cannot reach the model should return
something sensible rather than an error, and deciding what that fallback is — the previous
model, a heuristic, a popularity ranking — is a design decision the platform owns.
The lifecycle in the middle is best understood as a promotion pipeline with a rollback path.
Loading diagram…
The transition worth studying is Shadow to Challenger. Running a model on live traffic
without acting on its output is the cheapest way to discover that offline evaluation was
optimistic, and it is the step teams skip when they are in a hurry. The edge from Champion back
to Shadow is the other one to notice: it is drift detection closing the loop, and a platform
without that edge relies on somebody happening to check.
A maturity ladder, honestly described
Teams arrive in this area at very different starting points, and a great deal of interview
conversation is really about locating where a team sat and what the candidate moved. The ladder
below is descriptive rather than aspirational — most organisations are on the second or third
rung, and being there is not a failure.
Rung
What it looks like
What breaks
The next move
Notebook
A person runs a notebook and exports a file
Nobody can reproduce or rebuild it
Version the data and the code together
Scripted
Training is a script in a repository, run manually
Multi-tenancy, noisy neighbours, governance at scale
Consolidation and cost accountability per team
The rung that produces the most incidents is the third, because it looks finished. Training is
automated, artefacts exist, dashboards are present — and the decision to deploy is still a person
forming an opinion from a number, with no defined rollback. Interviewers probe this by asking who
decides that a new model goes live and what would make that decision reverse.
Who does this work
MLOps engineer and ML platform engineer are the two common titles, and they differ in scope: the
first is usually attached to a product team, taking that team's models to production and keeping
them there, and the second builds shared infrastructure that several modelling teams consume. The
second role's hardest problems are multi-tenancy problems — fair sharing of accelerated hardware,
per-team cost attribution, and a paved path that is genuinely easier than going around it.
Machine learning engineers straddle the boundary and in smaller organisations are the whole of it,
doing both the modelling and its deployment. That is common and it has a predictable failure mode:
the operational work is invisible until it is urgent, so it loses to modelling work in every
prioritisation until an incident reorders the queue. Data and platform engineers moving into ML
arrive with most of the required skills already — orchestration, containers, infrastructure as
code, pipeline reasoning — and need the specifics of training data correctness and drift. This is
one of the most reliable career transitions in the whole AI area, and it is under-advertised.
Site reliability engineers who take on ML systems bring the practices this field most lacks:
service level objectives, error budgets, incident review, and a professional scepticism about
dashboards nobody reads. Model risk and validation analysts, mostly in banking and insurance, are
a genuinely separate role — independent of the model's builder by design — who challenge and
approve models before deployment. Encountering that function without expecting it is a common
shock for engineers moving into a regulated employer.
A week in this work looks like this. You find out why a nightly training run failed and discover
an upstream schema change. You add a validation gate so the next one fails earlier and louder. You
investigate a latency regression that turns out to be cold starts after an autoscaling change. You
build the cost report that attributes GPU spend to three teams, and have an uncomfortable
conversation with the one that leaves notebooks running. You write the rollback runbook nobody has
needed yet. Very little of it is glamorous and most of it is the reason the models keep working.
What a prediction costs
Cost questions in this area are answered vaguely because people reason about instances rather than
predictions. Convert to unit cost and the architectural decision becomes obvious. The calculation
below carries units and states its assumptions, with no prices — because prices change and the
ratios do not.
The last block is the point. The most expensive decisions in an ML system are usually made during
modelling, by people who are not shown the bill, and an ML platform engineer who can trace a cost
back to a modelling decision and quantify it is doing the most valuable thing in the section.
Notice also that no vendor, hardware model or price appears, and the analysis still supports a
decision.
Demand, adoption and how that is changing
Demand is high, and the driver is a backlog rather than a novelty. A large number of organisations
now have models in production that were deployed as projects and are maintained by nobody in
particular: no reproducible training path, no monitoring beyond service uptime, no defined
retraining trigger, and no ability to answer why a specific decision was made. That accumulation
is what is being hired against. It is also why the roles skew towards experience — the work is
remediation of systems that already matter, which is a poor fit for a first job.
Three shifts are visible and worth stating carefully. The tooling has consolidated substantially,
which has moved the interview away from tool comparison and towards judgement: the question is now
less which registry you used and more what your promotion gate checked and what would make it
reject a model. Cost has become a first-class constraint rather than a finance concern, because
accelerated hardware spend is visible enough at board level that platform teams are asked to
defend it, which has made unit-cost reasoning a hiring topic. And governance is expanding beyond
financial services: model documentation, fairness testing and audit trails were a banking concern
and are becoming a general one, which raises the value of anyone who has operated under such a
regime.
The generative AI wave has had a double effect that is easy to misread. It has drawn attention and
budget away from classical ML platform work, making the area look quieter. At the same time it has
created a large new set of the same problems — versioning, evaluation gates, cost per request,
monitoring for behavioural change, provider risk — which is why LLMOps looks so much like MLOps
with different artefacts. Someone who understands promotion gates, drift and unit economics
transfers into that work readily. Someone whose expertise is one vendor's platform transfers less
well.
What makes it hard
Reproducibility is harder than it sounds because the data is part of the program. Pinning a
commit reproduces the code and not the result, since the training set was a query against a
mutable store, the environment has moved, a library changed a default, and some operations are
nondeterministic on accelerated hardware even with a fixed seed. Achieving genuine reproducibility
means versioning or snapshotting data, capturing the environment exactly, and recording the
sources of nondeterminism you chose to accept. Teams discover the gap at the worst possible moment
— when a regulator, a customer or an incident review asks how a specific decision was reached.
Point-in-time correctness is the deepest correctness problem in the section. A training row
timestamped in March must contain only what was knowable in March, and building that from stores
that hold current state is genuinely difficult. Slowly changing dimensions overwritten in place,
aggregates recomputed over the full history, late-arriving corrections applied retroactively, and
labels that were adjusted after the fact all conspire to leak the future into the past. The result
is a model that validates beautifully and fails in production, and unlike most production failures
there is no error to investigate — only a number that is lower than expected.
Drift must be detected without labels, and often without any ground truth at all. In many
applications the outcome arrives weeks or months later, and in some it never arrives for the cases
you acted on: you declined the loan, so you will never learn whether it would have defaulted. That
leaves proxy signals — input distribution shifts, changes in the distribution of predictions
themselves, a rise in cases near the decision boundary — each of which can move for benign
reasons. Building an alert that fires on real degradation and stays quiet through a seasonal
pattern or a marketing campaign is a genuinely hard engineering and statistical problem, and
"retrain monthly" is a schedule masquerading as an answer.
GPU cost is concentrated, spiky and easy to waste. Accelerated capacity is expensive enough
that a handful of instances can dominate a platform's bill, hard to share safely between workloads,
and prone to sitting idle: training jobs are bursty, inference is provisioned for peaks that occur
rarely, and development instances are left running. Meanwhile the highest-leverage cost decisions
were made upstream by modellers choosing an architecture and a candidate set size. Controlling this
requires attribution before optimisation, and attribution across shared hardware is itself an
engineering project.
Governance in regulated industries changes what a deployment is. Where model risk management
applies, a model cannot go live because its metrics improved. It requires documentation to a
prescribed standard, validation by someone independent of the developer, evidence of fairness
testing where protected characteristics are implicated, an explanation capability where decisions
affect individuals, and a retained audit trail. Engineers arriving from unregulated environments
consistently underestimate this, and the useful realisation is that the constraint is not
bureaucratic obstruction: it is the reason the pipeline, the registry and the reproducibility work
have to be real rather than aspirational.
Nothing fails loudly. This is the thread running through all of the above. A degraded model
returns a 200 response with a plausible number. A skewed feature produces slightly worse
predictions. A stale training set produces a model that is subtly wrong about a population that
recently changed. None of it pages anybody, which is why the practices here are preventative and
why they lose priority against work with visible urgency — right up until the loss is large enough
to be noticed by finance or by a regulator.
Why study it
Study it if you like operational engineering and want to work in machine learning without
competing for modelling roles. That combination is genuinely advantageous: the modelling entry
level is crowded, this one is not, and the skills — pipelines, containers, orchestration,
monitoring, cost — are ones a great many engineers already have most of. A data or platform
engineer can move into this work in months rather than years, and the transition is one of the
better-value moves available in the whole AI area.
Study it also if you want durable expertise. Model architectures churn and platform discipline does
not: reproducibility, promotion gates, drift detection and unit-cost reasoning have applied to
every generation of models including the current one, which is why LLMOps was not a new invention
so much as the same practices pointed at different artefacts. Someone who understands why a
promotion gate exists will still understand it after two more waves of models.
Be honest about who should not prioritise it. If what attracts you is the modelling itself —
choosing methods, engineering features, reasoning about metrics — you will find this section
frustrating, because it is deliberately about everything else, and
Machine Learning is the right place. If you have never operated a
production service, much of this will read as ceremony, because every practice here is the residue
of a specific failure and the practices only become convincing once you have seen one. And if you
want to build language-model product features, the LLMOps material in
Generative AI & Large Language Models is the faster route, with this
section as the deeper background.
Your first hour
Take any model you can train in a few minutes — anything at all, on any dataset — and make the
result reproducible. Then break the reproducibility on purpose and see what it takes to detect it.
This is a better first hour than installing a platform, because it teaches the thing the platform
exists to provide.
Train the model twice and diff the metrics. If they differ, find out why, and keep going until you
can explain every source of variation rather than merely suppressing it. Then write down a manifest
of everything a colleague would need to get your number.
The two NOT caught rows are the artefact and the lesson. They are the reason a data snapshot and
an environment record are not bureaucracy: without them, the same code and the same seed produce a
different model and nobody can say why. Being able to narrate that experiment — what you captured,
what you deliberately broke, and which breakage your manifest missed — is a stronger answer than a
list of tools you have configured.
If you have a second hour, add a validation gate. Write a check that refuses to train when the row
count moves by more than a stated fraction, or when a column's null rate jumps, and prove it by
corrupting the input. A pipeline that refuses bad data is worth more than one that trains on
schedule.
What this is not
It is not modelling. Nothing in this section will make a model more accurate, and a candidate who
has prepared modelling material for an MLOps loop has prepared for the wrong interview. The
relationship is the other way round: this discipline is what makes an accuracy claim believable and
durable, which is why the two roles need each other and why neither substitutes for the other.
It is not simply DevOps with a different noun. The pipeline, container and deployment craft
transfers, and then the differences begin: the artefact's behaviour depends on data you must also
version, promotion needs an evaluation gate rather than a passing test suite, a rollback restores a
model fitted to older data rather than restoring correct behaviour, and the system degrades while
sitting untouched. Those differences are the content of the section.
It is not a tool. A registry, a tracker, an orchestrator and a serving framework are how the
practices get implemented, and knowing their names is not knowing the practices. Interviewers
increasingly ask what your promotion gate checked and how you detected drift precisely because
those questions cannot be answered from a stack list.
It is not the same as LLMOps, although the overlap is large and growing. Versioning prompts against
a provider's weights, evaluating generated text, and managing a dependency on somebody else's
deprecation schedule are related problems with different artefacts, covered in
Generative AI & Large Language Models.
And it is not optional infrastructure that a team can add later. Every practice here is cheaper to
establish before a model matters than after, and the moment it starts to matter is exactly the
moment nobody has time. That asymmetry is the whole argument for the discipline, and it is the one
thing worth saying in an interview about why the work is worth funding.
A model does not fail loudly. It returns a plausible number for months while the world moves
away from the data it was fitted to, which is why this discipline is preventative or absent.
A ranking model, online serving, assumptions all stated
Peak traffic 600 requests persecond
Predictions per request 200 candidates scored
Peak prediction rate 120,000 predictions persecond
Throughput per accelerator 20,000 predictions persecond
Accelerators needed at peak 6, plus 1for headroom =7
Average traffic as share of peak 25per cent
Provisioned for peak, allday7 accelerators x 24 hours =168 unit-hours
Provisioned to follow demand roughly 60 unit-hours
Three readings of the same arithmetic
Static provisioning costs about 2.8x the demand-following equivalent
Cutting candidates from200to80 removes 60per cent of the compute outright
Caching repeated user-item pairs attacks the same term withno quality loss
The decision that dominates all three
Scoring 200 candidates online rather than 30 was a modelling choice with an
infrastructure bill attached, and nobody costed it at the time
run-2026-07-28-a
code git 4f2a9c1, clean working tree
data snapshot s3 path plus row count 184,203and a content hash
split bydate, train before 2026-04-01, test after
environment lockfile hash, python and library versions, CPU only
seeds numpy 42, framework 42, data shuffle 42
metric precisionat50=0.312
duration 4m 11s
Deliberate breakages, and whether the manifest catches them
Reorder the training rows caught only if the hash covers order
Rerun tomorrow against the live tableNOT caught without the snapshot
Change one library minor version caught by the lockfile hash
Train on a machine with an accelerator NOT caught - numerics differ