What has to be in place before you would let a model serve live traffic?

Can the feature be computed when the request arrives

Start here, because it disqualifies more models than any other check. Every feature the model consumes must be obtainable at the moment the decision is made, from data that exists at that moment, inside the latency budget of the call. A feature that took a warehouse query joining four tables is fine offline and unusable in a 40 ms request path.

The subtler half is temporal, and it is where leakage lives. Suppose the model scores a payment for fraud and one feature is "failed attempts by this card in the last hour". Built from a warehouse table assembled after the fact, that window may include attempts that happened after the payment you are scoring - because the table was written later and the query has no notion of when the decision occurred. The model learns from information it will not have, its offline metrics look excellent, and production performance is worse for reasons nothing in the evaluation explains.

The fix is point-in-time correctness: every training row is built by an as-of join keyed on the decision timestamp, so each feature reflects only what was knowable then. The much simpler alternative, when you can afford to wait, is to log the exact feature vector the serving path produced, attach the label when it arrives, and train on those logs. That guarantees temporal correctness by construction because the vector is the one that was really served.

Training-serving skew has separate causes and separate fixes

Treating skew as one hazard produces vague answers. It has at least three mechanisms.

The first is two implementations of one definition. The feature is SQL in the training pipeline and Python in the serving service, they agree on the day they are written, and then one of them is edited. Nothing fails; the model just receives a slightly different number than it was trained on. The fix is a single implementation used by both paths - which is the actual argument for a feature store, more than the caching is.

The second is different data behind the same definition. Training reads a settled warehouse table; serving reads a live API that sometimes times out and yields a default, or returns a null the training data never contained because the batch job backfilled it. A model trained without nulls has no defined behaviour for them, and whatever your framework does silently becomes model logic.

The third is representation: units, encodings, category vocabularies, and column order. A category the encoder has never seen at training time becomes an unknown bucket at serving time, and if that bucket is rare in training it is effectively untested.

Detection is a monitoring requirement rather than a code review requirement, because all three degrade quietly. Log served feature vectors, recompute the same features offline for a sample of those requests, and compare. A skew check that compares distributions but never compares the same request computed both ways will miss a definition that has diverged uniformly.

Version the model and the data as one artefact

A model is a build output of code, data, features and configuration. If you cannot reconstruct all four, you cannot explain a regression and you cannot reproduce the model you are being asked to defend. So the registry entry has to point at immutable identifiers, not descriptions.

# Illustrative shape, not any particular tool's schema.
model: checkout_fraud
version: 14
training_run: mlflow-run/9f2c1a
inputs:
  feature_definitions: git@a41f9c3        # the transformation code itself
  training_table: warehouse.fct_sessions@snapshot=4820193
  label_window: 2026-01-01/2026-05-31
  config: git@a41f9c3:configs/fraud_v14.yaml
serving:
  online_features: feature-store/checkout_v3   # same definitions, online path
  fallback: rules/checkout_v2                  # what serves if the model is down

training_table@snapshot=4820193 is doing the important work in that block. "The sessions table as of when I trained" is not a version; it is a description of a moment that has passed, and a mutable table will not give it back. Table formats such as Iceberg and Delta expose snapshot or commit identifiers exactly so a training run can name the bytes it read. The same applies to labels: a label set that was still being corrected when you trained will not reproduce either, so the label window and its extraction time both belong in the record.

Monitor in layers, because the label arrives last

Design monitoring around how quickly each signal becomes available, which is the constraint people leave out.

Operational signals arrive immediately: request latency, error rate, timeout rate on feature lookups, and the share of requests where a feature was null, defaulted or out of range. This layer catches the most common production failure, which is not a subtle statistical shift but a broken upstream job feeding a constant into the model.

Input distribution drift arrives within a window: compare each feature against a fixed reference sample from the training period, using a population-stability or two-sample test. Prediction drift arrives just as fast and needs less machinery - if the score distribution moves and the inputs did not, something in the serving path changed.

Realised performance arrives only after the label does, and the delay is a property of the business, not of your platform. Fraud chargebacks can take weeks; credit outcomes take months. Until then you have leading indicators and no ground truth, and being explicit about that in an interview is worth more than naming five drift tests.

Two things to say unprompted. Drift is not itself a defect - a marketing campaign changes your traffic mix, features shift, and the model may be perfectly fine - so a drift alert is a prompt to investigate, never an automatic retrain trigger. And running a test per feature across hundreds of features generates alerts constantly by chance, so either aggregate into a small number of monitored quantities or apply a multiple-comparison correction, otherwise the dashboard trains people to ignore it.

Rollback is a routing decision, and it includes the features

Rollback has to be achievable by changing which version receives traffic, with no retraining involved. Retraining under incident pressure is slow, and its output is a model nobody has evaluated - you would be replacing a known problem with an unknown one. So the previous version must remain deployed and servable, and the traffic split must be a configuration change.

Getting there requires the deployment path to support it: shadow mode first, where the new version scores real traffic and its predictions are logged but not used, then a canary on a slice of traffic, then full. Split the canary by a hash of a stable user identifier rather than per request, otherwise a returning user flips between versions and both your metrics and their experience become incoherent. And keep a non-model fallback - a rule set or a fixed decision - for the case where the model is unavailable rather than wrong, because "the service returns 500" is not an acceptable answer for a checkout path.

The part that catches people is the unit of rollback. Model version 14 was trained against feature definitions at commit a41f9c3. If the feature pipeline has since moved to b7d2e10 - a changed window, a re-encoded category, a renamed column filled with a default - then reverting the model alone gives you version 14 receiving inputs it was never trained on. You have not returned to a known-good state; you have created a combination that has never been evaluated anywhere, during an incident. The model and its feature transformations are one deployable unit, and the rollback plan has to say what happens to both.

Everything on this list is a version-control problem in disguise: the same feature computation on both paths, the dataset pinned by snapshot rather than by memory, and a rollback that moves the model and its features back together.