Everything between a model that works in a notebook and one a business depends on: reproducibility, feature and training infrastructure, registries, serving, drift, cost and governance.
24 published across 8 topics.
MLOps and ML platform fundamentals60 short answers on one page, for revising rather than studying.What has to be captured for a result to be reproducible, data and code versioning, seeds and nondeterminism, and why a notebook is not a record.
The prediction record must name the model version, the exact feature vector as served with its values, the identifier of the code that computed those features, the decision and any thresholds or overrides applied. Reconstruction after the fact is not possible, because feature pipelines and reference data have moved on.
Five things: the data version, the code commit, the resolved configuration, the pinned environment and the seeds. Even with all five, GPU kernels are often nondeterministic, so what you can promise is a statistically equivalent rerun and a recoverable artefact, not a bitwise identical one.
A notebook records the cells you wrote, not the order you ran them or the state they ran against, so the saved file frequently cannot reproduce its own outputs. Keep notebooks for exploration but promote anything you intend to defend into a parameterised script that logs its inputs to a tracking store.
Offline and online parity, point-in-time correctness, backfills, label collection and delay, and whether a feature store is worth its operational cost.
It earns its cost when several models share features and when you need offline and online parity plus feature history for correct backfills. It does not when one team runs one model, because you have added a stateful service to the request path for benefits nobody is drawing on yet.
Label latency caps your retraining cadence at roughly the label delay, forces evaluation to run on a lagging window that is always partially incomplete, and pushes you towards proxy labels that arrive sooner. Proxies buy speed at the price of optimising something adjacent to the outcome you care about.
A training row must contain only feature values that were knowable at its own event timestamp. A naive equality join on entity id attaches whatever the feature table holds now, including values derived from after the event, which leaks the future and produces offline metrics that collapse in production.
Turning a training script into a scheduled pipeline, data validation gates, distributed training, checkpointing, and failure recovery.
Split it at points where the intermediate output is worth keeping, make each stage idempotent so a retry is safe, put validation gates before training that fail the run loudly rather than warning, pass artefacts by immutable reference between stages, and set retry semantics per stage rather than globally.
Periodic checkpoints of model weights, optimiser state and data-loader position written atomically to durable storage; a resume path that is exercised rather than assumed; and interruption handling that checkpoints on the preemption signal. Each is justified by the expected cost of lost compute, not by principle.
Usually later than people reach for it. The first answer is a bigger single machine, because within one host the interconnect is far faster than any network and there is no distributed failure mode. Distribute when the model no longer fits one accelerator, or when a single-machine run is so long it blocks the work.
Promotion from staging to production, champion and challenger, automated gates on evaluation, rollback of a model, and shipping a model like code.
Gate on absolute thresholds, on a comparison against the incumbent evaluated on identical data, on per-slice regression checks so an aggregate gain cannot hide a subgroup loss, and on operational limits like latency and artefact size. Keep a human approval where a wrong decision is legally consequential.
Re-pointing traffic at the previous version is only safe if nothing else moved with it. The feature transformations, the input schema and any consumer calibrated to the current score distribution may have changed, so the unit of rollback is the model plus its features and its contract.
It is the system of record for which artefact is authoritative and why. It binds each version to the data snapshot, code commit and evaluation that produced it, records stage transitions and who approved them, and gives serving a stable identifier so any past prediction can be traced to the exact model that made it.
Batch, online and streaming inference, latency budgets, autoscaling and cold starts, GPU sharing, and shadow deployments.
Scale on queue depth or in-flight concurrency rather than request rate, because a GPU replica that takes minutes to load weights cannot be added after the queue has already formed. Then decide deliberately how much warm capacity to keep, since that is what you are paying to avoid cold starts.
Freshness decides it. If a prediction is still valid hours later, precompute it in a batch job and serve a lookup; if it depends on what the user did seconds ago, score it online; if it depends on an evolving event sequence, score it from a stream. Most mature systems end up hybrid.
Break the budget into its parts first, because feature fetching usually costs more than the model does. Then parallelise and cache the fetches, shrink the model with quantisation or distillation, set a timeout shorter than the budget, and define a degraded path that answers without the model.
Distinguishing data drift from concept drift, monitoring without labels, alerting on the right signal, and deciding when a retrain is warranted.
Data drift is a change in the distribution of the inputs; concept drift is a change in the relationship between inputs and the outcome. You can detect data drift from inputs alone, concept drift needs labels or a performance signal, and only concept drift necessarily invalidates what the model learned.
Retrain when there is evidence the learned relationship no longer holds and you have data that reflects the new one. A fixed schedule is a fallback for when you cannot measure that, not a decision, and a drift alert is as likely to mean an upstream pipeline broke as it is to mean the world moved.
Monitor what is available immediately: input distributions and data quality, the prediction distribution, and proxy outcomes that correlate with the label but arrive sooner. Do all of it per segment, because a small segment failing badly is invisible in an aggregate.
GPU economics, spot and preemptible capacity, right-sizing inference, caching, and knowing the unit cost of a prediction.
Measure utilisation first, because most fleets pay for idle accelerators rather than for too few. Then right-size to whatever each job is bound by, move interruptible work onto spot capacity behind tested checkpointing, and pool the fleet behind one queue instead of reserving per team.
Flat traffic and a tripled bill means cost per request tripled, so hunt what changed per request: a larger model version, more feature reads per prediction, retries multiplying calls, or warm capacity left running. Per-request cost attribution is what makes this answerable.
Divide the all-in hourly cost of the serving capacity by the predictions it delivers in that hour at your real utilisation, then add the per-request cost of feature reads. Utilisation dominates the result, and most of what people call cost per prediction is fixed cost that one more request does not change.
Model documentation and approval, fairness and bias testing, explainability where it is required, audit trails, and regulated-industry model risk.
A statement of intended and out-of-scope use, the provenance and lawful basis of the training data, evaluation results broken down by segment rather than only in aggregate, the known limitations and failure modes, and a signed approval record naming who accepted the risk against which evidence.
Establish who needs the explanation and what they will do with it, because a global account of model behaviour and a per-decision reason serve different needs. Attribution methods give you defensible local reasons within limits, but a post-hoc explanation is a second model of the first, not its reasoning.
Decide which groups you are testing across and how you will obtain that attribute, choose one fairness definition and justify it - because the common definitions are mathematically incompatible and cannot all be satisfied - then report performance and error rates per group with sample sizes and intervals attached.