MLOps and ML platform fundamentals

Because a notebook produces one artefact, on one machine, at one moment, with one operator present. A system produces a prediction for every request for years with nobody watching. Everything the notebook left implicit becomes a component you must own: where the input data comes from and on what schedule, how a feature is computed identically at training and serving time, where the artefact lives and how the serving tier chooses it, what happens when an upstream schema changes, and who is told when the predictions stop being good. The modelling code is usually the smallest part of the finished thing. The cost of not knowing that is a project that reports an excellent offline number for months and never ships, because the number was never what was blocking it.

A third input. Conventional delivery versions code and configuration; a machine learning system's behaviour is also determined by data, which changes continuously and without a commit. So the familiar guarantees weaken. The same code and the same pipeline produce a different model next Tuesday. Tests cannot assert an exact output, only a bound on a distribution. A regression can arrive with no deployment at all. What MLOps adds is versioning and validation for that third input, plus monitoring of a quality signal that arrives later than the prediction it judges. The cost is a heavier pipeline than a service of comparable size, which is why teams resist paying it until the first silent degradation teaches them why.

Each stage produces an artefact the next one needs, and each has a characteristic way of failing quietly.

stage                artefact produced          fails as
-------------------  -------------------------  ----------------------------------
data availability    versioned snapshot         trains on data the serving
                                                path cannot supply
feature definition   one spec, both paths       train-serve skew
training pipeline    run record plus model      a result nobody can reproduce
evaluation           metrics against a          a number with nothing to
                       named baseline             compare it to
registry, promotion  a named production         nobody can say what is live
                       version
serving              a prediction inside        timeouts rather than wrong
                       the latency budget         answers
consumption          a decision that changes    accurate model, no effect on
                                                  anything
monitoring           a quality signal           degradation found by a customer

Read the right-hand column rather than the left. Every failure listed is silent: none of them raises an exception, and most of them leave the offline metric looking fine. That is the structural difference from a service, where the failures announce themselves as errors, and it is why the controls in this discipline are assertions and monitors rather than tests and reviews.

The two rows teams skip are the last two. A model that nobody's decision depends on is a research artefact with a deployment, and it is remarkably common — the score is served, the operational team keeps using their old rule, and nothing changes. Consumption is a stage, with its own integration work and its own threshold to agree.

The ordering also tells you where to start. Data availability comes first because it is the only stage that can invalidate all the others: if a feature cannot be computed at serving time within your budget, then no amount of modelling work downstream is recoverable. Establish the serving path for a trivial model before you tune anything.

The data, and it changes in two ways that look identical from outside. Upstream producers alter what they emit — a new enum value, a column that begins arriving null, an amount whose units changed from pence to pounds — and the world itself moves, so the same honest measurement now means something different. Neither appears in a diff, neither fails a build, and both change your predictions. What follows is that a machine learning system needs input assertions where a service has request validation: schema, ranges, cardinality and null rates checked at the boundary on every run, with the run failing rather than proceeding. Treating data as trusted because it comes from inside the company is the assumption that breaks.

A pipeline is one model's path from data to deployed artefact. A platform is the set of shared components that lets the tenth pipeline be built in a day rather than a quarter: a feature layer, a training-job runner, a registry, a serving runtime, and monitoring every model gets by default. The distinction matters because someone always proposes building the platform for the first model, which is a mistake — you have not yet learned which parts generalise, so you will abstract the wrong ones. Build two or three pipelines, extract what they actually duplicated, and accept that a platform's real cost is a team committed to maintaining it indefinitely.

Entanglement first: a model fits all its inputs jointly, so changing one feature changes the effective weight of every other, and there is no such thing as a local change. Then undeclared consumers — teams that discovered your prediction table and now depend on it, which you find out when you rescale the score. Then feedback loops, where the model's own output shapes the data it is later trained on, so it converges on its own past preferences. Then configuration sprawl, since most of a mature pipeline's surface area is thresholds, windows and flags rather than logic. None of the four is caught by a unit test, which is why review is the wrong control here and monitoring is the right one.