MLOps and ML platform fundamentals

Because a notebook produces one artefact, on one machine, at one moment, with one operator present. A system produces a prediction for every request for years with nobody watching. Everything the notebook left implicit becomes a component you must own: where the input data comes from and on what schedule, how a feature is computed identically at training and serving time, where the artefact lives and how the serving tier chooses it, what happens when an upstream schema changes, and who is told when the predictions stop being good. The modelling code is usually the smallest part of the finished thing. The cost of not knowing that is a project that reports an excellent offline number for months and never ships, because the number was never what was blocking it.

A third input. Conventional delivery versions code and configuration; a machine learning system's behaviour is also determined by data, which changes continuously and without a commit. So the familiar guarantees weaken. The same code and the same pipeline produce a different model next Tuesday. Tests cannot assert an exact output, only a bound on a distribution. A regression can arrive with no deployment at all. What MLOps adds is versioning and validation for that third input, plus monitoring of a quality signal that arrives later than the prediction it judges. The cost is a heavier pipeline than a service of comparable size, which is why teams resist paying it until the first silent degradation teaches them why.

Each stage produces an artefact the next one needs, and each has a characteristic way of failing quietly.

stage                artefact produced          fails as
-------------------  -------------------------  ----------------------------------
data availability    versioned snapshot         trains on data the serving
                                                path cannot supply
feature definition   one spec, both paths       train-serve skew
training pipeline    run record plus model      a result nobody can reproduce
evaluation           metrics against a          a number with nothing to
                       named baseline             compare it to
registry, promotion  a named production         nobody can say what is live
                       version
serving              a prediction inside        timeouts rather than wrong
                       the latency budget         answers
consumption          a decision that changes    accurate model, no effect on
                                                  anything
monitoring           a quality signal           degradation found by a customer

Read the right-hand column rather than the left. Every failure listed is silent: none of them raises an exception, and most of them leave the offline metric looking fine. That is the structural difference from a service, where the failures announce themselves as errors, and it is why the controls in this discipline are assertions and monitors rather than tests and reviews.

The two rows teams skip are the last two. A model that nobody's decision depends on is a research artefact with a deployment, and it is remarkably common — the score is served, the operational team keeps using their old rule, and nothing changes. Consumption is a stage, with its own integration work and its own threshold to agree.

The ordering also tells you where to start. Data availability comes first because it is the only stage that can invalidate all the others: if a feature cannot be computed at serving time within your budget, then no amount of modelling work downstream is recoverable. Establish the serving path for a trivial model before you tune anything.

The data, and it changes in two ways that look identical from outside. Upstream producers alter what they emit — a new enum value, a column that begins arriving null, an amount whose units changed from pence to pounds — and the world itself moves, so the same honest measurement now means something different. Neither appears in a diff, neither fails a build, and both change your predictions. What follows is that a machine learning system needs input assertions where a service has request validation: schema, ranges, cardinality and null rates checked at the boundary on every run, with the run failing rather than proceeding. Treating data as trusted because it comes from inside the company is the assumption that breaks.

A pipeline is one model's path from data to deployed artefact. A platform is the set of shared components that lets the tenth pipeline be built in a day rather than a quarter: a feature layer, a training-job runner, a registry, a serving runtime, and monitoring every model gets by default. The distinction matters because someone always proposes building the platform for the first model, which is a mistake — you have not yet learned which parts generalise, so you will abstract the wrong ones. Build two or three pipelines, extract what they actually duplicated, and accept that a platform's real cost is a team committed to maintaining it indefinitely.

Entanglement first: a model fits all its inputs jointly, so changing one feature changes the effective weight of every other, and there is no such thing as a local change. Then undeclared consumers — teams that discovered your prediction table and now depend on it, which you find out when you rescale the score. Then feedback loops, where the model's own output shapes the data it is later trained on, so it converges on its own past preferences. Then configuration sprawl, since most of a mature pipeline's surface area is thresholds, windows and flags rather than logic. None of the four is caught by a unit test, which is why review is the wrong control here and monitoring is the right one.

Five things, and most teams keep three. The code, at a commit, with a clean tree. The data, as a version or a query with a pinned cut-off rather than a table name. The configuration — hyperparameters, feature list, split definition and label window. The environment, as a fully resolved dependency set or an image digest, not a requirements file with open ranges. And the seeds, for every library that draws random numbers, including the data loader's shuffle. Miss one and a rerun differs, which destroys the only thing an experiment exists to do: attribute a change in the metric to the change you made. The working test is whether last quarter's model can be rebuilt today and produce identical predictions.

Because its cells can run in any order and its state survives edits to the code that produced it. The output under a cell may have come from a version that no longer exists there, a variable can hold a value from a definition you deleted, and the saved file records the last render rather than an execution. So a notebook displaying a metric is a claim, not evidence. It is also effectively unversionable, since the diff carries outputs and execution counts. Use notebooks for exploration, then move anything whose result you intend to quote into a script that runs top to bottom in a fresh process and logs its own inputs, and treat the notebook's number as provisional until it has.

One run, written by the training code rather than by a person afterwards.

run 4f2a91   status finished   duration 41m
  git          9c1de07   working tree clean: true
  data         orders@snapshot-2026-06-30   rows 4,182,905
  features     spec v7   32 features   list digest 5c0e...
  split        time-based   train < 2026-05-01   valid 05-01..06-01
  params       depth 8   lr 0.05   rounds 900   seed 20260630
  environment  image digest 1c7f...   accelerator driver pinned
  metrics      valid pr_auc 0.412   recall at 1% fpr 0.276
  slices       age_under_25 0.388   channel_app 0.421   new_customer 0.301
  artefacts    model.bin   feature_importance.json   eval_slices.csv
  parent       run 3d81b0   changed: rounds 600 -> 900

The two fields that make this a record rather than a log are the last ones. The parent link with an explicit diff means the run answers a question — did more rounds help — instead of merely reporting a number, and a chain of such links is the actual history of the model. The clean-tree flag is what stops the whole thing being fiction, because a metric produced from uncommitted code cannot be reproduced by anyone including its author.

A spreadsheet fails at all of this for one reason: it is written by hand, after the fact, by someone who already knows which run they liked. The runs that get recorded are the good ones, the fields that get filled are the ones that changed, and the data version is omitted because it felt constant. Automatic capture from inside the training job is the entire mechanism.

The slice row is worth insisting on early. Storing per-segment metrics at training time costs nothing and is the only way a later promotion gate can compare a candidate against a champion on the segments that matter, rather than on a single average that hides them.

The cost is instrumentation discipline. Every parameter must reach the tracker from the code path that used it, not from a config file that may have been edited since, and any value the training job derives at runtime — a resolved date, a sampled row count — has to be logged as an output rather than assumed from input.

Several places, and the seed closes only the first. Library randomness — initialisation, shuffling, dropout, subsampling — is seeded and behaves. Parallel reduction is not: floating-point addition is not associative, so summing gradients across eight workers in a different order each run produces a slightly different result that compounds. Accelerator kernels are chosen by autotuning, and the fastest kernel for a shape can vary between runs or driver versions. Data is a source whenever the training set is a query against a live table. And early stopping amplifies a tiny difference into a different model. Determinism is achievable but not free, since deterministic kernels are slower, so decide whether you need bit-identical runs or only statistically indistinguishable ones.

Not by copying it. Three options, in increasing order of what they cost and guarantee. Version by query and cut-off, which is correct only on a genuinely append-only source whose late corrections you can bound. Version by table-format snapshot, where the storage layer retains a file manifest per commit and you record the snapshot identifier, giving an exact rerun for the price of kept metadata and undeleted files. Or materialise the training set itself, which is small next to the source and is the only version that survives the source being restructured or purged. The unacceptable answer is a table name with no time reference, because it is indistinguishable from having no version at all.

The distinction is between declaring what you want and recording what you got.

declared, not pinned              recorded, reproducible
--------------------------------  ------------------------------------------
scikit-learn                      lock file with every transitive
numpy>=1.24                         dependency at an exact version
python:3.12                       base image referenced by digest
pip install at container start    dependencies baked into the image
accelerator driver from the host  driver and runtime versions logged
                                    with the run

The left column is what most training repositories contain, and each line fails differently. An open version range resolves to whatever was published most recently, so a minor release that changes a default — how a class weight is interpreted, how ties are broken — silently alters your model. A tag is mutable, so the same tag pulls different bytes next month. Installing at container start means the build is done on a machine with network access at an arbitrary moment, which is both irreproducible and a supply-chain surface.

The driver line is the one that catches teams who did everything else correctly. Numerical results on an accelerator depend on the runtime and driver, so an image pinned by digest running on a re-provisioned node fleet can produce a different result — and because nothing in the repository changed, the investigation starts in the wrong place. Record the host-side versions as run metadata even though you cannot pin them from the image.

The cost is real and worth naming: larger images, a rebuild whenever you want a security patch, and a lock file that someone must deliberately refresh. In exchange, the difference between two runs is only what you intended to change, which is the precondition for any claim about a model.

A model version is the artefact's identity: these bytes, this name, this number. Lineage is the graph behind it — which run produced it, from which data snapshot, under which feature spec and commit, and which evaluation it passed. Versioning only the artefact tells you what is running and nothing about why it was allowed to run, so it cannot answer the questions that actually get asked: was this customer's record in the training set, were these two models compared on the same split, which upstream change explains the shift in scores. Lineage has to be emitted by each stage as it executes, which is why it cannot be reconstructed afterwards and why retrofitting it is nearly always a rebuild.

That every feature value in a training row was computable at that row's own timestamp, from data which existed then. It is what makes an offline evaluation predictive of online behaviour, because at serving time you have no access to the future. It is harder than it sounds because most warehouse tables hold current state: a customer row carries today's address and today's risk band, with no record of March's. Point-in-time correctness therefore needs either event-sourced history or valid-from and valid-to columns on every dimension you join, and getting it wrong produces the most demoralising outcome in this discipline — a model that scores brilliantly offline and mediocrely in production, with no bug to find.

The same query twice, against a dimension that keeps history and one that does not.

-- Leaky. `customers` holds current state, so a January transaction is
-- joined to a risk band that was set in July -- possibly set *because*
-- of the very fraud this row records.
SELECT t.txn_id, t.amount, c.risk_band, t.is_fraud
  FROM transactions t
  JOIN customers    c ON c.customer_id = t.customer_id
 WHERE t.txn_ts >= '2026-01-01';

-- Correct. The dimension is versioned, and the join selects the version
-- that was in effect at the transaction's own timestamp.
SELECT t.txn_id, t.amount, c.risk_band, t.is_fraud
  FROM transactions   t
  JOIN customer_history c
    ON c.customer_id = t.customer_id
   AND t.txn_ts >= c.valid_from
   AND t.txn_ts <  c.valid_to
 WHERE t.txn_ts >= '2026-01-01';

The same rule applies to any aggregate, and there the trap is subtler:

-- The window must be bounded by the row's own timestamp, and must
-- exclude the row itself -- otherwise the count includes the event
-- being predicted.
SELECT t.txn_id,
       ( SELECT count(*)
           FROM transactions p
          WHERE p.customer_id = t.customer_id
            AND p.txn_ts <  t.txn_ts
            AND p.txn_ts >= t.txn_ts - INTERVAL '30 days' ) AS txns_30d
  FROM transactions t;

What the first query taught the model is that a raised risk band predicts fraud, which is true and useless, because in production the band is raised after the fraud rather than before. The tell is a large gap between offline and online performance with every input distribution matching — the features are fine, their timestamps are not.

Current-state tables are the default because they are what operational systems naturally produce, so leakage of this kind is the normal condition rather than an unusual mistake. Fixing it means asking the owning system to record when each value became true, and that request usually arrives too late to help the first model.

The self-exclusion in the aggregate is the detail candidates omit. A window defined as "the last thirty days" evaluated on a training row includes the row's own transaction, so a feature meant to describe prior behaviour partly describes the event itself. It is a small leak with a large effect on a rare-event model, and it survives review because the SQL reads correctly.

The cost of doing this properly is history tables, larger storage, and joins that are considerably more expensive than an equality join. Pay it, and check the offline-to-online gap as the assurance that you did.

Timing, mostly. Training reads a settled warehouse table while serving reads a stream that is minutes behind, so a "count in the last hour" feature is systematically lower at serving time. Restatement is another: the offline source has been corrected since, so training sees the tidy figure and production sees the original. Then availability — a feature present for ninety-nine per cent of training rows and missing for ten per cent of live requests, because the upstream call times out under load. And default handling, where a missing value becomes zero at serving but a dropped row in training. All four are invisible unless you log the vector that was actually scored and compare its distribution to the training set's.

Because you cannot learn from or measure an outcome you do not yet have. If a charge-off is confirmed only after ninety days, then the freshest trustworthy label describes a decision made three months ago, and retraining weekly means retraining on nearly identical data each time — motion without information. The delay fixes three things simultaneously: the earliest date a deployed model can be evaluated, the useful retraining interval, and the length of the window during which a broken model runs entirely unmeasured. Shortening the delay, usually with a correlated early proxy, buys more than any modelling change available to you. The proxy's cost is that it is not the label, so it can be gamed by the model.

Work backwards from today, and the windows fall out of the delay.

today = 2026-07-28          label maturity = 90 days

usable label horizon    events on or before 2026-04-29
validation window       2026-03-01 .. 2026-04-29    mature, held out
training window         2023-05-01 .. 2026-02-28
blind period            2026-04-30 .. 2026-07-28    labels not yet known

what follows
  the freshest signal the model can learn from is 90 days old
  a model deployed today is first measurable on 2026-10-26
  three months of live traffic will be scored with no ground truth

The blind period is the number to put in front of a sponsor, because it is the answer to "how quickly would we know if this went wrong" and the answer is three months. Everything else in the monitoring design follows from accepting that, which is why label delay belongs in the first design conversation rather than in the monitoring one.

The tempting shortcut is to include the partly matured recent rows, and it introduces a bias that is easy to miss. Among events not yet at maturity, the bad outcomes that have already been confirmed are the ones that go bad fastest, so labelling early oversamples quick failures and teaches the model that slow failures are good. Either wait for maturity or model the time to event explicitly; do not mix.

During the blind period you monitor what exists — input distributions, prediction distribution, decision rates — and manufacture what does not, by holding back a small random control that bypasses the model and by sampling decisions for human review. Those two are the only sources of unconfounded signal you will have for three months, and both must be running from day one because neither can be created retrospectively.

One definition and two access paths, which is not the same as two products. The offline path answers "give me this feature as of a million historical timestamps", a large point-in-time join that is indifferent to latency. The online path answers "give me this feature for one entity, now", in single-digit milliseconds. No storage engine is good at both, so the shape is a warehouse or lake on one side and a key-value store on the other, kept in agreement by materialising both from the same computation. What a feature store product sells is that agreement plus a catalogue; what it costs you is another stateful system sitting on the critical serving path.

It depends on reuse, not on sophistication. With two models and features owned by the team that built them, a shared SQL module plus a materialisation job gives you parity for a fraction of the weight. It starts to pay when several teams consume the same features, when point-in-time joins are being hand-written repeatedly and inconsistently, or when low-latency lookups are needed anyway and somebody is about to build a bespoke cache for them. Count the costs honestly: a new serving dependency that can take your model down, backfill jobs to run and monitor, a registry that goes stale, and one more place for a definition to diverge from the code that uses it.

You train on a version of history that never existed at decision time. Restatements are routine in financial and operational data, so a backfill run today hands the model the settled figure while production will see the provisional one — the same shape as leakage, and just as flattering offline. The rule is that a backfill must reproduce what was knowable then, which requires the source to record when a value became true as well as what it was. Where it cannot, say so in the model's documentation, expect the offline metric to be optimistic, and size the expected online drop rather than being surprised by it.

Everything about failure. A script is run by a person who reads the traceback; a pipeline runs at three in the morning and must decide for itself whether its inputs are acceptable, where partial work goes, and what a retry means. So the script acquires explicit stages with typed inputs and outputs, idempotent writes so a rerun cannot double-count, a validation gate before the expensive step, and an artefact store in place of a local path. It also acquires a schedule, which raises the question a script never had to answer: what should happen when yesterday's run has not finished and today's is due. Answer that deliberately, because the default is usually two runs writing to the same place.

Six stages, and the gates matter more than the stages.

flowchart LR
    A[Ingest snapshot<br/>pin the data version] --> B[Validate schema<br/>and distributions]
    B --> C[Build features<br/>point-in-time join]
    C --> D[Train<br/>with checkpoints]
    D --> E[Evaluate<br/>overall and by slice]
    E --> F[Register candidate<br/>if gates pass]

Validation sits second on purpose. It is the cheapest stage and the one most likely to reject, so putting it before feature building and training means a bad upstream feed costs you seconds rather than hours of accelerator time. The general rule is to order stages by cost ascending wherever the dependencies allow, and this pipeline has exactly one such freedom.

The first node's job is to pin, not to copy. It resolves "the orders table" to a snapshot identifier and writes that identifier into the run record, so every downstream stage and every later investigation refers to the same immutable input. A pipeline that reads a live table at each stage can have two stages disagree within one run.

The last node is a registration, not a deployment, and that boundary is the point of the whole diagram. The pipeline's authority ends at producing a candidate with its evidence attached; deciding that the candidate should receive traffic is a separate flow with different gates and, in most organisations, a human. Merging the two gives you a system that can promote a model trained on a bad Tuesday.

What the DAG deliberately omits is a branch for "metrics were slightly worse". Evaluation records the numbers and the gate decides; the pipeline does not negotiate with itself. Keeping the judgement out of the DAG is what makes the gate auditable later.

Four families. Schema: expected columns present, types unchanged, no silently added category. Volume: row count inside a band around the expected, because a half-empty snapshot from a broken upstream job trains a confident, wrong model. Distribution: per-feature null rate, cardinality, mean and quantiles against the last accepted run, which is what catches unit changes and stuck sensors. Relational sanity: join hit rates, duplicate keys, and labels present for the expected share of rows. The gate's value is failing cheaply before hours of compute; its cost is calibration, because thresholds set too tight make the pipeline the noisiest system you own and get it muted.

Because a model will train on anything. Hand it a snapshot missing a third of its rows and it converges, produces a plausible metric, and passes a gate that compares it against a baseline computed on the same bad data. The failure is silent by construction and surfaces weeks later as a decision-quality problem, with the pipeline having reported green throughout. A hard failure costs a delayed model and someone's evening; a soft failure costs a wrong model with no signal that anything happened. That asymmetry is the whole argument for gates that abort, and it is why "the DAG was green" is a statement about the orchestrator and never about the data.

When the model or its activations will not fit on one accelerator, or when a single-device run is long enough to block iteration. Below that, one larger device beats several smaller ones, because you avoid the entire class of problems distribution introduces. Data parallelism is the cheap case — replicate the model, split the batch, all-reduce the gradients — and it costs communication proportional to parameter count on every step, plus an effective batch size that invalidates your learning-rate schedule. Model and pipeline parallelism are for models that do not fit at all, and cost implementation effort and idle devices at the stage boundaries. Nodes also fail more often in aggregate, which makes checkpointing mandatory rather than prudent.

The ability to lose an hour instead of a week. A twelve-hour job on preemptible capacity will be interrupted, and without checkpoints each interruption restarts from zero, so the expected completion time can exceed the nominal run length without bound. Checkpoints also make a run inspectable and let you resume from before a divergence rather than retuning blind. The costs are storage and stall time, since writing state pauses progress, and that sets the interval: frequent enough that expected lost work is small relative to the write overhead. The detail usually missed is that a resumable checkpoint needs optimiser state and the data iterator's position, not only the weights.

By making every stage's output addressable and every write idempotent, so rerunning stage four neither requires redoing stages one to three nor leaves two copies of anything. In practice that means outputs keyed by run identifier and data version rather than by wall-clock date, writes that replace a partition instead of appending to it, and stages able to detect their own completed work and skip it. Retry assumes the failure was transient; recovery assumes it was not, and lets you fix a bug and resume from the point of failure. The cost is storage for intermediate artefacts and the discipline never to write to a mutable location from inside a pipeline.

Identity, state and provenance. A bucket holds bytes at a path, so "which model is in production" is answered by convention and somebody's memory. A registry names a model, numbers its versions, records which version currently occupies which stage, keeps the run and data lineage behind each version, and stores the evaluation it passed together with who approved it. That turns promotion into an auditable transition rather than a file copy, and rollback into a stage reassignment rather than an archaeology exercise. The cost is that the registry becomes authoritative, so every deployment path must read from it — one process that side-loads an artefact makes the registry a record of intentions rather than of facts.

The gates are code, the thresholds were agreed before the run, and one of them fails.

candidate v118 vs production champion v112        live traffic share: 0

gate                      rule                              result
------------------------  --------------------------------  ----------------
lineage complete          data version, commit and env      pass
                            all recorded
input schema              identical to champion, or a       pass
                            declared breaking change
headline metric           pr_auc >= champion + 0.005        pass
                                                              0.412 vs 0.404
slice regression          no named segment worse than       FAIL
                            champion by more than 0.010       age_under_25
                                                              0.388 vs 0.402
calibration               mean predicted vs observed        pass
                            rate within 5% per decile
latency p99               <= 40 ms at twice peak rps        pass  31 ms
prediction distribution   mean score within 10% of          pass
                            champion on shadow traffic
owner approval            named sign-off recorded           pending

outcome: blocked. The headline metric improved while a protected
segment regressed, which is precisely what a single number hides.

The slice gate is the one that earns the policy its existence. An average improvement is compatible with a meaningful regression in any segment small enough not to move it, and the segments that are small in the data are routinely the ones that are large in a regulator's view. Naming the segments in advance also stops the argument that happens after a failure, when someone proposes that this particular slice was never important.

Calibration deserves its own line because a model can rank better and still break everything downstream. If the score distribution shifts, every fixed threshold built on it now means a different volume of approvals or reviews, so an improved model can flood an operations team. Rank quality and probability quality are separate properties and only one of them is what the business consumes.

Latency belongs here rather than in serving, because a candidate that cannot meet the budget is not a candidate. Measuring it at twice peak rather than at current peak is the cheap way to avoid promoting a model that is fine until the next sale.

The cost of a policy like this is flakiness. A gate that fails intermittently gets a manual override, the override becomes routine, and the policy is then decorative. So each gate needs a deterministic evaluation set and a threshold derived from observed run-to-run variation rather than chosen to look strict.

Each hop answers a different question, and they are not interchangeable.

flowchart LR
    T[Training run<br/>produces candidate] --> R[Registry<br/>version recorded]
    R --> G[Automated gates<br/>metrics and slices]
    G --> S[Shadow traffic<br/>no user impact]
    S --> C[Canary<br/>small live share]
    C --> P[Production<br/>champion replaced]
    P --> B[Previous version<br/>kept reselectable]

The gates ask whether the model is good on history. Shadow asks whether the system works on reality — real input distributions, real missing values, real latency under real concurrency — and it can answer that with no user exposure at all, because the output is logged and discarded. Skipping shadow is how a model that passed every offline gate fails on the first request containing a category the test set never had.

The canary asks the only question shadow cannot: what happens when the prediction changes what someone experiences. That requires a sticky split per entity so one customer does not oscillate between models, and the serving version stamped on every prediction so the outcomes can be attributed when they eventually arrive.

The final node is the one people leave out of the diagram and regret. The previous champion stays registered, its image stays pullable, and its input schema stays satisfiable by the current feature pipeline — because rollback is only fast if the old version is still deployable. Once promotion is a configuration change against the registry rather than a rebuild, both directions take seconds.

A fixed comparison and a fixed population. A bare threshold drifts, because next month's evaluation set is easier or harder, so the gate must compare candidate against current champion on the same rows, with the split defined by the pipeline rather than chosen by the author. It needs per-slice results, since an average improvement routinely conceals a regression that matters commercially or legally. It needs a calibration check, because a model whose ranking improved while its probabilities shifted breaks every downstream threshold. And it needs its thresholds committed before the run, or the gate stops being a gate and becomes a negotiation held by whoever wants to ship.

Because offline evaluation answers a narrower question than it appears to. It scores a fixed historical set under training-time feature definitions, so it cannot see the serving path's timing, its missing-value rates or its latency. It cannot see the effect of the model's own output on behaviour, which means a ranking model that looks better on logged interactions may only be better at predicting what the previous model chose to show people. And it says nothing about the decision layer, where a better score at an unchanged threshold can change volumes enough to break an operational team. Shadow answers the first gap, a live experiment answers the second, and the third needs a conversation rather than a metric.

By selecting the previous version, not by rebuilding it. That requires four preconditions: the earlier artefact still exists, its environment image is still pullable, its input schema is still satisfiable by the current feature pipeline, and the serving layer takes its version from configuration rather than from a baked image. Where all four hold, rollback is a configuration change measured in seconds. Where any fails, "roll back" means a training run, which is hours at best. The subtlety is that a rollback does not undo the predictions already served or the downstream state they created, so a runbook needs a remediation step alongside the switch.

A live alert, and seven steps in the order they must happen.

trigger  decline rate 3.1% against a 1.4% baseline, sustained 20 minutes

 1  freeze     stop the retraining schedule and any in-flight promotion
 2  identify   registry: production = v118 since 09:14, previous = v112
 3  diagnose   is this the model or its inputs?
                 check feature null rates and upstream job status first
                 -- a broken feed is not fixed by rolling back a model
 4  switch     point the production alias at v112
                 configuration change, no build, no deploy
 5  verify     decline rate and p99 latency back inside bounds within
                 10 minutes; predictions carry model_version so the
                 change is visible in the data, not just in a dashboard
 6  contain    list every decision made by v118 between 09:14 and now;
                 queue them for re-scoring or manual review
 7  record     mark v118 blocked in the registry with the reason, so
                 tomorrow's promotion cannot select it again

deliberately absent: retraining. That is the fix, not the mitigation.

Step three is the one that gets skipped under pressure, and skipping it is how a team rolls back a perfectly good model while a null-filled feed keeps degrading the old one just as badly. The symptom of a broken input and the symptom of a bad model are identical from the alert's point of view, so the check has to be in the runbook rather than left to judgement at speed.

Step six is what separates a model runbook from a service runbook. Restoring a service stops the harm; restoring a model stops new harm and leaves an hour of wrong decisions standing, each of which may already have declined a customer or released a payment. Someone must own reversing them, and that is only possible if predictions were logged with enough input to re-score.

Step seven exists because promotion is automated. Without an explicit block, the next scheduled run can compare v118 favourably on the metric that did not catch the problem and put it straight back. The registry is the only place that decision is durable.

The precondition for the entire runbook is step four being a configuration change. If rollback requires a build, the numbers above become hours and the containment list grows accordingly, which is the argument for version selection at runtime.

Because the slow, expensive, non-deterministic part is the part you most want to test. A service's tests are fast and exact; a training pipeline's meaningful test costs accelerator time and returns a distribution. So model CI layers instead: fast unit tests on feature transforms and inference code with fixed inputs and expected outputs, a smoke training run on a tiny sample proving the pipeline executes end to end, contract tests on the serving interface, and the full evaluation as a promotion gate rather than a commit gate. Making evaluation a pull-request check turns every commit into an hour's wait and teaches the team to bypass it.

By what the prediction depends on and when it is consumed. If it depends only on slowly changing state and is read later, precompute in batch: no serving tier, no latency budget, retryable failures, and by a wide margin the lowest cost per prediction. If it depends on the request itself — this basket, this amount, this session — it has to be online. Streaming sits between them, scoring as events arrive and writing the result for later reading, which fits features that must be fresh and decisions that need not be synchronous. Start batch and move only the part that genuinely needs the request, because each step up multiplies the operational surface you must run.

Start from the product requirement and spend downwards.

requirement: 95th percentile checkout risk decision <= 120 ms

  network in and out, caller to service       12 ms
  request parsing and validation               3 ms
  online feature fetch, 4 keys, batched       28 ms
  feature assembly and encoding                6 ms
  model forward pass, batch of one            22 ms
  decision rules and response construction     4 ms
  -----------------------------------------------
  accounted                                   75 ms
  headroom                                    45 ms

spending the headroom on resilience
  one retry of the feature fetch            +28 ms  -> 103 ms, inside
  a second retry                            +28 ms  -> 131 ms, breaches

so the policy is one retry, then serve with the feature marked missing
-- which is only safe if the model was trained with that feature
sometimes absent.

Adding component 95th percentiles overstates the total, because the components rarely peak on the same request, so the 75 ms figure is conservative and the real headroom is larger. That conservatism is deliberate: a budget you can defend is worth more than a tight estimate, and the arithmetic is only used to decide what you can afford, not to predict a measurement.

The feature fetch dominates, which is the usual finding and points at the fix. Fewer keys per prediction, a colocated store rather than one across a network boundary, and precomputing anything that does not depend on the request will all move that line, whereas a faster model moves a smaller one. Optimise the largest row, not the most interesting one.

The retry policy is the part that reaches back into training. A degraded response requires the model to behave sensibly with a missing value, and that only happens if missingness appeared in the training data with the same encoding used at serving time. Deciding the timeout policy at serving time, after training, is how a resilience feature becomes a correctness bug.

The budget also has to be a test rather than a document. Assert the accounted total in a load test at the traffic you claim to support, and treat any new dependency in the path as requiring a line in the table before it is merged.

Three sequential costs on a fresh replica: pulling the image, loading weights into host or device memory, and warming up — early requests hitting uncompiled kernels, empty caches and an unallocated memory pool. On a large model the weight load dominates and is roughly size over storage bandwidth. What removes it is keeping capacity warm, not scaling faster: images pre-pulled on the node, weights on a local volume or baked into the image, a readiness probe that passes only after a synthetic inference, and a replica floor that never reaches zero. Scale-to-zero and a tight latency objective are incompatible, and the honest trade is paying for idle capacity.

Because the unit is coarse, slow and expensive. A new accelerator node takes minutes to become available, may not be available in your zone at all, and is charged whether or not it computes, so reactive scaling arrives after the traffic it was meant to serve. The signal is also misleading: reported device utilisation is a poor proxy for saturation, so you scale on queue wait or batch latency instead. What follows is that accelerator capacity is planned rather than autoscaled — a warm floor sized to normal peak, a queue that sheds or degrades under overload, and a scale-down far slower than you would use for stateless web replicas.

By splitting on the prediction, not on the deployment. A container canary asks whether the new binary is healthy; a model canary asks whether the new model's decisions are acceptable. That needs the traffic split to be sticky per entity, so one customer does not oscillate between two models and produce an incoherent experience; the serving version stamped on every prediction, so outcomes can be attributed weeks later; and comparison on decision-level metrics rather than error rates. The awkwardness is that the business metric usually matures after the canary would have ended, so a model canary runs on proxies: prediction distribution, decision-rate shift, and coverage of every slice.

It trades one against the other deliberately. Accelerators are throughput devices, so scoring thirty-two requests together costs little more than scoring one, and cost per prediction falls sharply. The cost is that each request waits either for the batch to fill or for the window to expire, which adds queueing delay to every response including the first. A dynamic batcher is therefore configured by two numbers — maximum batch size and maximum wait — and the wait must be derived from the latency budget rather than from the throughput you would like. Under light traffic the wait is pure loss, which is why batching helps a busy service and harms a quiet one.

As close to the model as possible while keeping a single definition. The common mistake is having callers assemble the vector, which spreads feature logic into every consumer and guarantees divergence inside a year. The common overcorrection is a separate feature service behind another network hop, which adds its own tail latency and its own outage to a path that had neither. The workable middle is a feature client library inside the scoring service, reading a colocated low-latency store, with definitions generated from the same specification the training job consumed — one definition, one hop, and a documented degraded mode for when the store is unreachable.

By where the change shows up. Data drift moves the inputs, so feature distributions shift against the training reference while the mapping from features to outcome may be entirely intact — and you see it immediately, without labels. Concept drift leaves the inputs looking familiar and changes what they mean, so every distribution test passes and performance falls only once labels mature. The diagnosis is therefore ordered: feature distributions first, prediction distribution second, performance on mature labels third. Inputs shifted and performance held means drift you can note and ignore; inputs stable and performance fallen means the relationship changed, and no amount of data engineering addresses that.

Eight signals, and only three of them are allowed to wake anybody.

signal                  method                  warn     page     action
----------------------  ----------------------  -------  -------  ---------------
feature null rate       vs 7-day baseline       +2 pp    +10 pp   upstream check
feature distribution    PSI vs training ref     0.10     0.25     investigate feed
input schema            exact match             --       any      block scoring
                                                          change
prediction mean score   vs 7-day baseline       5%       15%      threshold review
decision rate           approvals per hour      10%      25%      business
                                                                    escalation
slice coverage          requests per segment    --       segment  new population
                                                          absent
performance             pr_auc, mature labels   -0.010   -0.030   retrain candidate
                          only
serving p99 latency     vs the objective        80% of   breach   capacity
                                                  budget

paging rows: input schema, decision rate, serving latency. Everything
else opens a ticket, because distribution movement is information and
not an incident.

Every threshold here has to be derived rather than chosen. Run the check against the last few months of history, observe how much the signal moves on days when nothing was wrong, and set the warn level above that band — otherwise you have encoded a guess as an alert and the first week will teach everyone to mute it.

The prediction-distribution row is the highest-value cheap signal on the list. It needs no labels, no reference data beyond your own recent history, and it moves for almost every real problem: a broken feature, a new population, a bad promotion. If you can afford exactly one monitor, it is this one.

Slice coverage is the row nobody builds and everybody needs. A segment that simply stops appearing usually means an upstream filter changed or a channel broke, and it is invisible in aggregate distributions because the remaining traffic looks normal. It is also the earliest warning that the population you trained on no longer exists.

The cost of this table is per-model tuning and the ongoing work of keeping it honest. Eight signals across twenty models is a hundred and sixty thresholds, which is why the platform should provide the checks and the defaults, and the model owner should only override where their model differs.

The branch is the whole diagram, and it is the step that gets automated away.

flowchart LR
    M[Monitor<br/>features and labels] --> D{Signal type}
    D -->|inputs broken| U[Fix upstream<br/>no retrain]
    D -->|distribution moved| A[Assess<br/>is performance hurt]
    A --> T[Train on a<br/>fresh window]
    T --> G[Gates and<br/>shadow traffic]
    G --> P[Promote or reject]

Most drift alerts land on the upper branch. A feed that started sending nulls, a producer that changed units, a join that quietly stopped matching — all of them present as drift, and all of them are made permanent rather than fixed by retraining, because the new model learns the fault as though it were the world. Classifying the signal before acting on it is the difference between a loop that helps and a loop that launders bugs into models.

The assess step exists because inputs move constantly without harming anything. Retraining on every distribution shift means retraining continuously, each run consuming compute and each promotion carrying risk, in exchange for no measured improvement. The question to answer before triggering is whether performance on mature labels has actually degraded, and whether the fresh window contains enough of the new regime to learn from.

The promote node stays a decision, not an edge. A loop that trains and promotes without a gate is a system that can degrade itself unattended overnight, and the data it trains on is exactly the data that just triggered an alert. Automate the detection and the training; keep a gate, and for a material model keep a person, on the promotion.

You monitor everything upstream of the label and accept you are measuring stability rather than quality. Input distributions and null rates catch upstream breakage. Prediction distribution catches the model behaving differently on a new population. Decision rates catch business impact directly and are often the most sensitive signal available. Then you manufacture partial ground truth: sample a small fraction of decisions for human review, hold back a small random control that bypasses the model, and instrument any weak downstream signal that correlates with the outcome. Without at least one of those, the model's quality is unfalsifiable, which is a governance problem before it is a technical one.

Page for things both urgent and actionable by the person you wake: the serving tier missing its objective, a schema violation meaning predictions are being made on wrong inputs, a feature store outage, or a decision-rate shift large enough to have commercial consequences within hours. Ticket everything statistical — distribution movement, slow metric decay, a new segment appearing — because the correct response takes days of analysis that nobody performs at three in the morning. Getting this backwards has a predictable ending: drift alerts fire weekly, get muted as noise, and the one that mattered arrives invisibly among them.

Usually not, and answering yes reflexively is the most common mistake in this area. Ask first whether the pipeline is broken, because a feed sending nulls or a unit that changed presents identically to drift, and retraining bakes the fault in permanently. Ask second whether performance has actually degraded on mature labels, since inputs move constantly without hurting anything. Ask third whether the recent data contains enough of the new regime to learn from — retraining on a fortnight of a shifted world can be worse than keeping the old model. Retrain when the relationship has genuinely changed and you hold the labels to learn the new one.

Because it replaces the model before anybody has to explain why replacement was needed. A weekly retrain masks slow degradation, so nobody notices that a feature has been dead for a month; it also changes what is in production without a decision, so a bad week's data can be promoted while everyone assumes the schedule is a safety measure. Scheduling is defensible where labels arrive quickly and the world genuinely moves, and only with the same gates a manual promotion would face. What is never defensible is a schedule with automatic promotion and no slice comparison, which is a system with standing permission to degrade itself.

The arithmetic is trivial; the assumptions are the entire exercise, so state every one.

assumptions
  one accelerator instance costs                     C per hour
  it sustains, at the configured batch size          900 predictions/second
  target utilisation, leaving headroom for peak      60%

derived
  effective throughput      900 x 0.60            =  540 per second
  predictions per hour      540 x 3600            =  1,944,000
  compute cost per prediction                     =  C / 1,944,000

then the lines people omit
  online feature fetch, 4 keys per prediction      -> store cost per prediction
  logging prediction plus input vector             -> often the largest line
                                                       for a small model
  the warm floor kept for cold starts              -> paid at zero requests
  the shadow copy during evaluation                -> doubles compute while
                                                       it runs
  training and retraining                          -> amortised over the
                                                       predictions served
                                                       before the next retrain

sanity check
  if a prediction is worth less than its cost, the answer is a nightly
  batch table, a cheaper model, or no model at all

The utilisation assumption dominates everything below it. At sixty per cent you pay for roughly one and two-thirds instances per instance's worth of work; at twenty per cent, which is a common real figure for a fleet sized to peak, the cost per prediction is three times the naive calculation. Anyone quoting a unit cost without stating utilisation is quoting a best case that the fleet has never achieved.

The omitted lines routinely exceed the compute line for smaller models. Logging every prediction with its feature vector is not optional — it is what makes rollback containment, skew detection and audit possible — but it is a write per prediction into retained storage, and at high volume that is a larger bill than the inference. Budget it deliberately rather than discovering it.

The amortisation line is where the retraining cadence conversation becomes financial. If a training run costs many hours of accelerator time and is amortised over a week of predictions, a daily schedule multiplies that line sevenfold for whatever improvement daily retraining actually delivers, which is usually unmeasured.

The sanity check is the only line that changes decisions. Comparing cost per prediction against value per prediction ends more inappropriate real-time projects than any architecture review, and it is arithmetic anyone can do in the first week rather than the last.

Idle time, not throughput. A device is charged by the hour whether or not it is computing, so the bill is set by how many device-hours you hold and how little of each you use — and serving utilisation is typically low, because the fleet is sized for peak and peak is brief. What follows is that the largest savings are structural rather than algorithmic: consolidate several small models onto one device, batch aggressively where the budget allows, move anything delay-tolerant to batch scoring on cheaper capacity, and cache repeated inputs. Buying a faster device to fix a twenty-per-cent-utilised fleet increases the bill and improves nothing.

For anything checkpointed and rerunnable, which covers most training, batch scoring and hyperparameter search: an interruption costs the work since the last checkpoint, and the discount is usually large enough that the arithmetic favours it even with frequent reclamation. It is not safe for the serving tier of a latency-sensitive path, because reclamation removes replicas at short notice and the replacement pays a cold start. The pattern that works is a small on-demand floor for serving plus spot for everything asynchronous, with jobs written to expect interruption rather than to hope. The hidden cost is availability: spot for a scarce device type may not exist when you need it.

Measure the real request mix, then fit the deployment to the tail latency you must meet at forecast peak rather than to the mean. Establish single-replica capacity under realistic input sizes first, because throughput on variable-length inputs bears little relation to throughput on a fixed benchmark. Choose the smallest device that holds the model plus its activation memory, since a large device at low utilisation costs more per prediction than two smaller ones. Set the replica floor from cold-start time and the ceiling from your capacity quota. Then revisit after any change to batching, quantisation or model size, because each invalidates the measurement that justified the shape.

The intended use and the uses explicitly ruled out, the training data with its known gaps, the evaluation including per-slice results rather than one headline number, the definitions of the metrics, the known failure modes, the monitoring in place, and the named owner. It is signed by whoever accepts the consequences of the model being wrong, which is a business owner rather than the person who trained it — and the signature is the mechanism, because it forces somebody with authority to read the limitations. The cost is staleness: a card describing a model two retrains ago is worse than none, so it must be generated from the run record rather than written by hand.

Whichever definition you chose, and the choice is a decision rather than a discovery. Equal error rates across groups, equal selection rates and equal calibration are mutually incompatible except in degenerate cases, so no model satisfies all three and a claim of fairness without a named criterion is empty. Testing therefore means agreeing the protected attributes, the criterion and the tolerated gap in advance, then measuring per slice on mature labels with enough volume per group for the number to mean anything. The costs are needing an attribute you may not be permitted to collect, and accepting that closing a gap usually reduces the headline metric.

Wherever a decision must be justified to the person it affected or to a regulator: credit and insurance declines, employment screening, and anything carrying a statutory right to an explanation. There the requirement is a specific faithful reason for one decision, which is a much stronger demand than global feature importance and rules out any post-hoc approximation you could not defend under challenge. That is the real argument for an inherently interpretable model on those paths even at some cost in accuracy — the explanation is part of the product rather than documentation about it. Elsewhere explainability serves debugging and reviewer trust, and a local attribution method is adequate.

That a particular decision was produced by a particular model version, from particular inputs, under an approval in force at that time. So every prediction is logged with its model version, its input vector or a hash of it, a timestamp and the decision taken; every model version is tied to its data snapshot, commit and evaluation; and every promotion records who approved it against which gates. The test is whether you can reconstruct one customer's decision from eighteen months ago and show why it was permitted. Retention and storage are the cost, and logging the input vector is the part most often omitted and most often needed.

It inserts independent validation between building a model and using it. A second team that does not report to yours reviews conceptual soundness, tests the data and assumptions, reproduces your results and challenges the choice of model — and it can refuse. Practically, reproducibility and documentation stop being hygiene and become the deliverable; models are inventoried and tiered by materiality so the heavy process lands where it matters; and retraining is itself a change requiring approval. The cost is cycle time measured in months, which is exactly why the platform work that makes any model reproducible on demand pays for itself here before anywhere else.

A strong answer starts with the label. It says when ground truth for those six weeks becomes available and therefore whether the question can be answered at all yet, which immediately separates people who have operated a model from people who have trained one. It then names the signals that exist in the meantime — input distributions, prediction distribution, decision rate, slice coverage — and is explicit that these are leading indicators and performance on mature labels is the confirmation. It mentions that predictions were logged with their model version, so outcomes can be attributed rather than assumed, and names the source of unconfounded signal: a held-back control or a review sample the model did not influence. Finally it says what each signal would cause: which one triggers investigation, which one triggers rollback, and why a retrain is not the first response to any of them. A weak answer substitutes infrastructure for measurement. It describes dashboards, monitoring stacks and alerting tools without naming a single number that would change anyone's mind, or it quotes the offline metric from before deployment as though it were a statement about today.