MLOps & ML Platform

What this area actually covers

This section is about the gap between a model that produced a good number in a notebook and a model a business depends on. Closing that gap means being able to reproduce the result, having training data that reflects only what was knowable at the time, running training as a scheduled pipeline rather than a script somebody remembers how to invoke, promoting a version through gates that can reject it, serving predictions inside a latency budget on hardware you can afford, detecting that the world has moved before your users do, and being able to answer an auditor's question about a decision made eighteen months ago.

None of that is modelling. That is the defining characteristic of the discipline and the reason it is a separate section: the skills are software engineering, infrastructure, data engineering and operations, applied to an artefact with two properties ordinary software does not have. The first is that its behaviour is determined by data as much as by code, so versioning the code version is not enough to reproduce anything. The second is that it degrades without changing — the code is identical, the deployment is untouched, and the model gets worse because the world it was fitted to has moved. Every unusual practice in this section descends from one of those two facts.

The boundary with adjacent work is worth drawing precisely, because the roles are hired separately. Choosing a model family, engineering features for predictive power, selecting a metric and reasoning about leakage is Machine Learning. Building the pipelines that populate a warehouse or lake is data engineering, in Data & AI Engineering. Serving and evaluating language models has its own operational discipline covered as LLMOps within Generative AI & Large Language Models — the concepts here transfer, but the artefacts differ, because there you version prompts and configuration against somebody else's weights rather than versioning your own.

On this site's structure: the older Data & AI Engineering section still carries an MLOps subsection from an earlier authoring wave, and it has not been removed. Treat it as the compressed introduction to what this section covers across eight subsections.

Two things are wrongly bundled in. The first is "MLOps is DevOps for models", which is true enough to be misleading: the pipeline, container and deployment craft does transfer directly, and the parts that do not transfer — point-in-time correct training data, evaluation gates, drift detection without labels, and the fact that a rollback restores code but not necessarily behaviour — are exactly the parts that make the job difficult. The second is tool knowledge as a substitute for the discipline. Naming an experiment tracker, an orchestrator and a registry describes a stack; explaining what has to be captured for a result to be reproducible describes the understanding those tools exist to encode.

The eight areas underneath

The order follows a model's life. The first two are what you need before training is meaningful, the next two are how a model gets built and promoted, then how it serves, then how you find out it has stopped working, and the last two are the constraints that decide whether the whole arrangement is sustainable — money and permission. Reproducibility is first because everything else is unverifiable without it.

Experiment Tracking & Reproducibility

What must be captured for a result to be recreatable: the code version, the data version, the configuration, the environment, the random seeds, and the hardware where it affects numerics. It covers data and code versioning together, sources of nondeterminism that survive a fixed seed, and the reason a notebook is a workspace rather than a record. It is first because it is the foundation of every claim anyone makes in this field: without it, "the new model is better" is an assertion about two runs that differed in ways nobody recorded. Interviewers ask what you would need to reproduce a six-month-old result, and the strong answer includes the data snapshot and the environment, not just a commit hash.

Feature Stores & Training Data

Parity between the features computed for training and those computed at serving time, point-in-time correctness so a training row contains only what was knowable at its timestamp, backfilling a new feature over history, collecting labels and coping with the delay before they arrive, and the honest question of whether a feature store is worth the operational cost for a given team. It is separate because this is where the hardest correctness problem in the section lives. Train-serve skew produces a model that validated well and underperforms in production for reasons no monitoring will explain, and the cause is almost always that two pieces of code computed the same feature differently.

Training Pipelines & Orchestration

Converting a training script into a scheduled pipeline with dependencies, data validation gates that stop a run when the input is wrong rather than training on it, distributed training when a single machine is insufficient, checkpointing, and recovering from a failure partway through a long job. Its own subsection because the transition from script to pipeline is where most of the engineering rigour enters, and because the validation gate is the highest-value component: a pipeline that trains happily on a day when an upstream job silently produced nulls will deploy a worse model on schedule and tell nobody.

Model Registry & CI/CD

Promotion from development to staging to production, champion and challenger arrangements, automated gates that compare a candidate against the incumbent on a fixed evaluation set, rolling a model back, and treating a model version as a deployable artefact with a change history. Separate because the rules differ from code deployment in a way that matters: a code rollback restores previous behaviour deterministically, whereas rolling back a model returns you to a version fitted to older data, which may itself now be wrong. Expect material on what a promotion gate should actually check, which is more than a single metric threshold.

Serving & Inference Infrastructure

Batch, online and streaming inference and how to choose; latency budgets and where the model sits within one; autoscaling and the cold-start problem when a container must load large weights; sharing GPUs between workloads; and shadow deployments that run a new model on live traffic without acting on its output. It exists as its own area because serving is where ordinary distributed systems engineering meets the specific awkwardness of model artefacts — large, slow to load, occasionally requiring specialised hardware — and because the choice of serving mode is frequently the decision that makes a project affordable or not.

Monitoring, Drift & Retraining

Distinguishing data drift, where the inputs have changed distribution, from concept drift, where the relationship between inputs and outcome has changed; monitoring quality when labels arrive weeks late or never; choosing a signal worth alerting on rather than one that fires constantly; and deciding whether a retrain is warranted or whether the problem is upstream. This is the subsection that has no analogue in conventional operations, and it contains the field's hardest open question: how to tell that a model has degraded when you cannot yet observe whether it was right. Proxy signals, prediction distribution shifts, and deliberate holdouts are the tools, and each has limits worth knowing precisely.

Cost & Capacity Management

The economics of accelerated hardware, using interruptible capacity for work that tolerates it, right-sizing inference rather than provisioning for a peak that occurs weekly, caching predictions where inputs repeat, and being able to state the unit cost of a single prediction. It is separate because cost in machine learning systems is unusually concentrated and unusually visible: a small number of instances can dominate a platform bill, and an ML platform team is routinely asked to defend that line. Being able to express a model's cost per thousand predictions, and to say which architectural choice drives it, is a differentiating skill.

Governance & Model Risk

Model documentation, formal approval before deployment, fairness and bias testing, providing explanations where a regulation or a customer requires them, audit trails that can reconstruct a past decision, and the model risk management practice established in financial services and now spreading. It closes the section because in regulated industries it is the gate everything else must pass. It is also the subsection that most changes what an interview looks like: a bank or an insurer will ask how a model was validated by someone independent of its builder, and a candidate who has only worked where deployment was an engineering decision will not have an answer.

Where it sits in a real system

An ML platform sits between the data platform and the product, and it takes a dependency in each direction that a normal service does not. Upstream it depends not merely on a data system being available but on its semantics being stable: a column whose meaning changes, a pipeline that starts emitting nulls, a definition adjusted for a reporting need, will each degrade a model without producing a single error anywhere. That is why data validation gates sit inside training pipelines rather than being left to the data team.

Downstream it serves a product decision, which means a prediction has a consumer with an expectation attached. Latency budgets arrive from that consumer. So does the requirement that the system degrade usefully: a recommendation service that cannot reach the model should return something sensible rather than an error, and deciding what that fallback is — the previous model, a heuristic, a popularity ranking — is a design decision the platform owns.

The lifecycle in the middle is best understood as a promotion pipeline with a rollback path.

stateDiagram-v2
    [*] --> Experiment
    Experiment --> Candidate: run reproducible and logged
    Candidate --> Shadow: passes evaluation gate
    Shadow --> Challenger: live traffic, output not acted on
    Challenger --> Champion: wins on the online metric
    Champion --> Retired: replaced by a successor
    Challenger --> Candidate: loses, back for rework
    Champion --> Shadow: drift detected, successor trialled
    Retired --> [*]

The transition worth studying is Shadow to Challenger. Running a model on live traffic without acting on its output is the cheapest way to discover that offline evaluation was optimistic, and it is the step teams skip when they are in a hurry. The edge from Champion back to Shadow is the other one to notice: it is drift detection closing the loop, and a platform without that edge relies on somebody happening to check.

A maturity ladder, honestly described

Teams arrive in this area at very different starting points, and a great deal of interview conversation is really about locating where a team sat and what the candidate moved. The ladder below is descriptive rather than aspirational — most organisations are on the second or third rung, and being there is not a failure.

The rung that produces the most incidents is the third, because it looks finished. Training is automated, artefacts exist, dashboards are present — and the decision to deploy is still a person forming an opinion from a number, with no defined rollback. Interviewers probe this by asking who decides that a new model goes live and what would make that decision reverse.

Who does this work

MLOps engineer and ML platform engineer are the two common titles, and they differ in scope: the first is usually attached to a product team, taking that team's models to production and keeping them there, and the second builds shared infrastructure that several modelling teams consume. The second role's hardest problems are multi-tenancy problems — fair sharing of accelerated hardware, per-team cost attribution, and a paved path that is genuinely easier than going around it.

Machine learning engineers straddle the boundary and in smaller organisations are the whole of it, doing both the modelling and its deployment. That is common and it has a predictable failure mode: the operational work is invisible until it is urgent, so it loses to modelling work in every prioritisation until an incident reorders the queue. Data and platform engineers moving into ML arrive with most of the required skills already — orchestration, containers, infrastructure as code, pipeline reasoning — and need the specifics of training data correctness and drift. This is one of the most reliable career transitions in the whole AI area, and it is under-advertised.

Site reliability engineers who take on ML systems bring the practices this field most lacks: service level objectives, error budgets, incident review, and a professional scepticism about dashboards nobody reads. Model risk and validation analysts, mostly in banking and insurance, are a genuinely separate role — independent of the model's builder by design — who challenge and approve models before deployment. Encountering that function without expecting it is a common shock for engineers moving into a regulated employer.

A week in this work looks like this. You find out why a nightly training run failed and discover an upstream schema change. You add a validation gate so the next one fails earlier and louder. You investigate a latency regression that turns out to be cold starts after an autoscaling change. You build the cost report that attributes GPU spend to three teams, and have an uncomfortable conversation with the one that leaves notebooks running. You write the rollback runbook nobody has needed yet. Very little of it is glamorous and most of it is the reason the models keep working.

What a prediction costs

Cost questions in this area are answered vaguely because people reason about instances rather than predictions. Convert to unit cost and the architectural decision becomes obvious. The calculation below carries units and states its assumptions, with no prices — because prices change and the ratios do not.

A ranking model, online serving, assumptions all stated
  Peak traffic                            600 requests per second
  Predictions per request                 200 candidates scored
  Peak prediction rate                    120,000 predictions per second
  Throughput per accelerator              20,000 predictions per second
  Accelerators needed at peak             6, plus 1 for headroom = 7
  Average traffic as share of peak        25 per cent
  Provisioned for peak, all day           7 accelerators x 24 hours = 168 unit-hours
  Provisioned to follow demand            roughly 60 unit-hours

Three readings of the same arithmetic
  Static provisioning costs about 2.8x the demand-following equivalent
  Cutting candidates from 200 to 80 removes 60 per cent of the compute outright
  Caching repeated user-item pairs attacks the same term with no quality loss

The decision that dominates all three
  Scoring 200 candidates online rather than 30 was a modelling choice with an
  infrastructure bill attached, and nobody costed it at the time

The last block is the point. The most expensive decisions in an ML system are usually made during modelling, by people who are not shown the bill, and an ML platform engineer who can trace a cost back to a modelling decision and quantify it is doing the most valuable thing in the section. Notice also that no vendor, hardware model or price appears, and the analysis still supports a decision.

Demand, adoption and how that is changing

Demand is high, and the driver is a backlog rather than a novelty. A large number of organisations now have models in production that were deployed as projects and are maintained by nobody in particular: no reproducible training path, no monitoring beyond service uptime, no defined retraining trigger, and no ability to answer why a specific decision was made. That accumulation is what is being hired against. It is also why the roles skew towards experience — the work is remediation of systems that already matter, which is a poor fit for a first job.

Three shifts are visible and worth stating carefully. The tooling has consolidated substantially, which has moved the interview away from tool comparison and towards judgement: the question is now less which registry you used and more what your promotion gate checked and what would make it reject a model. Cost has become a first-class constraint rather than a finance concern, because accelerated hardware spend is visible enough at board level that platform teams are asked to defend it, which has made unit-cost reasoning a hiring topic. And governance is expanding beyond financial services: model documentation, fairness testing and audit trails were a banking concern and are becoming a general one, which raises the value of anyone who has operated under such a regime.

The generative AI wave has had a double effect that is easy to misread. It has drawn attention and budget away from classical ML platform work, making the area look quieter. At the same time it has created a large new set of the same problems — versioning, evaluation gates, cost per request, monitoring for behavioural change, provider risk — which is why LLMOps looks so much like MLOps with different artefacts. Someone who understands promotion gates, drift and unit economics transfers into that work readily. Someone whose expertise is one vendor's platform transfers less well.

What makes it hard

Reproducibility is harder than it sounds because the data is part of the program. Pinning a commit reproduces the code and not the result, since the training set was a query against a mutable store, the environment has moved, a library changed a default, and some operations are nondeterministic on accelerated hardware even with a fixed seed. Achieving genuine reproducibility means versioning or snapshotting data, capturing the environment exactly, and recording the sources of nondeterminism you chose to accept. Teams discover the gap at the worst possible moment — when a regulator, a customer or an incident review asks how a specific decision was reached.

Point-in-time correctness is the deepest correctness problem in the section. A training row timestamped in March must contain only what was knowable in March, and building that from stores that hold current state is genuinely difficult. Slowly changing dimensions overwritten in place, aggregates recomputed over the full history, late-arriving corrections applied retroactively, and labels that were adjusted after the fact all conspire to leak the future into the past. The result is a model that validates beautifully and fails in production, and unlike most production failures there is no error to investigate — only a number that is lower than expected.

Drift must be detected without labels, and often without any ground truth at all. In many applications the outcome arrives weeks or months later, and in some it never arrives for the cases you acted on: you declined the loan, so you will never learn whether it would have defaulted. That leaves proxy signals — input distribution shifts, changes in the distribution of predictions themselves, a rise in cases near the decision boundary — each of which can move for benign reasons. Building an alert that fires on real degradation and stays quiet through a seasonal pattern or a marketing campaign is a genuinely hard engineering and statistical problem, and "retrain monthly" is a schedule masquerading as an answer.

GPU cost is concentrated, spiky and easy to waste. Accelerated capacity is expensive enough that a handful of instances can dominate a platform's bill, hard to share safely between workloads, and prone to sitting idle: training jobs are bursty, inference is provisioned for peaks that occur rarely, and development instances are left running. Meanwhile the highest-leverage cost decisions were made upstream by modellers choosing an architecture and a candidate set size. Controlling this requires attribution before optimisation, and attribution across shared hardware is itself an engineering project.

Governance in regulated industries changes what a deployment is. Where model risk management applies, a model cannot go live because its metrics improved. It requires documentation to a prescribed standard, validation by someone independent of the developer, evidence of fairness testing where protected characteristics are implicated, an explanation capability where decisions affect individuals, and a retained audit trail. Engineers arriving from unregulated environments consistently underestimate this, and the useful realisation is that the constraint is not bureaucratic obstruction: it is the reason the pipeline, the registry and the reproducibility work have to be real rather than aspirational.

Nothing fails loudly. This is the thread running through all of the above. A degraded model returns a 200 response with a plausible number. A skewed feature produces slightly worse predictions. A stale training set produces a model that is subtly wrong about a population that recently changed. None of it pages anybody, which is why the practices here are preventative and why they lose priority against work with visible urgency — right up until the loss is large enough to be noticed by finance or by a regulator.

Why study it

Study it if you like operational engineering and want to work in machine learning without competing for modelling roles. That combination is genuinely advantageous: the modelling entry level is crowded, this one is not, and the skills — pipelines, containers, orchestration, monitoring, cost — are ones a great many engineers already have most of. A data or platform engineer can move into this work in months rather than years, and the transition is one of the better-value moves available in the whole AI area.

Study it also if you want durable expertise. Model architectures churn and platform discipline does not: reproducibility, promotion gates, drift detection and unit-cost reasoning have applied to every generation of models including the current one, which is why LLMOps was not a new invention so much as the same practices pointed at different artefacts. Someone who understands why a promotion gate exists will still understand it after two more waves of models.

Be honest about who should not prioritise it. If what attracts you is the modelling itself — choosing methods, engineering features, reasoning about metrics — you will find this section frustrating, because it is deliberately about everything else, and Machine Learning is the right place. If you have never operated a production service, much of this will read as ceremony, because every practice here is the residue of a specific failure and the practices only become convincing once you have seen one. And if you want to build language-model product features, the LLMOps material in Generative AI & Large Language Models is the faster route, with this section as the deeper background.

Your first hour

Take any model you can train in a few minutes — anything at all, on any dataset — and make the result reproducible. Then break the reproducibility on purpose and see what it takes to detect it. This is a better first hour than installing a platform, because it teaches the thing the platform exists to provide.

Train the model twice and diff the metrics. If they differ, find out why, and keep going until you can explain every source of variation rather than merely suppressing it. Then write down a manifest of everything a colleague would need to get your number.

run-2026-07-28-a
  code            git 4f2a9c1, clean working tree
  data            snapshot s3 path plus row count 184,203 and a content hash
  split           by date, train before 2026-04-01, test after
  environment     lockfile hash, python and library versions, CPU only
  seeds           numpy 42, framework 42, data shuffle 42
  metric          precision at 50 = 0.312
  duration        4m 11s

Deliberate breakages, and whether the manifest catches them
  Reorder the training rows                       caught only if the hash covers order
  Rerun tomorrow against the live table            NOT caught without the snapshot
  Change one library minor version                caught by the lockfile hash
  Train on a machine with an accelerator           NOT caught - numerics differ

The two NOT caught rows are the artefact and the lesson. They are the reason a data snapshot and an environment record are not bureaucracy: without them, the same code and the same seed produce a different model and nobody can say why. Being able to narrate that experiment — what you captured, what you deliberately broke, and which breakage your manifest missed — is a stronger answer than a list of tools you have configured.

If you have a second hour, add a validation gate. Write a check that refuses to train when the row count moves by more than a stated fraction, or when a column's null rate jumps, and prove it by corrupting the input. A pipeline that refuses bad data is worth more than one that trains on schedule.

What this is not

It is not modelling. Nothing in this section will make a model more accurate, and a candidate who has prepared modelling material for an MLOps loop has prepared for the wrong interview. The relationship is the other way round: this discipline is what makes an accuracy claim believable and durable, which is why the two roles need each other and why neither substitutes for the other.

It is not simply DevOps with a different noun. The pipeline, container and deployment craft transfers, and then the differences begin: the artefact's behaviour depends on data you must also version, promotion needs an evaluation gate rather than a passing test suite, a rollback restores a model fitted to older data rather than restoring correct behaviour, and the system degrades while sitting untouched. Those differences are the content of the section.

It is not a tool. A registry, a tracker, an orchestrator and a serving framework are how the practices get implemented, and knowing their names is not knowing the practices. Interviewers increasingly ask what your promotion gate checked and how you detected drift precisely because those questions cannot be answered from a stack list.

It is not the same as LLMOps, although the overlap is large and growing. Versioning prompts against a provider's weights, evaluating generated text, and managing a dependency on somebody else's deprecation schedule are related problems with different artefacts, covered in Generative AI & Large Language Models.

And it is not optional infrastructure that a team can add later. Every practice here is cheaper to establish before a model matters than after, and the moment it starts to matter is exactly the moment nobody has time. That asymmetry is the whole argument for the discipline, and it is the one thing worth saying in an interview about why the work is worth funding.

A model does not fail loudly. It returns a plausible number for months while the world moves away from the data it was fitted to, which is why this discipline is preventative or absent.