Design the gate that decides whether a newly trained model gets promoted.

Score both models in the same run, on the same data

Every other part of the gate depends on this. The comparison that matters is not "the challenger scored 0.87" against "the incumbent's recorded score was 0.86". Those two numbers were produced on different evaluation sets, at different times, possibly with different preprocessing. The recorded score of the incumbent is a historical fact about a historical dataset and it is not a baseline.

So the evaluation stage loads both artefacts and scores both on one freshly constructed, temporally held-out set, in the same process, through the same feature code. That is the only comparison a promotion decision can rest on, and it also happens to catch a class of bug nothing else does: if the incumbent's score on today's data has quietly collapsed, you have discovered a production problem while evaluating a candidate.

Four families of criteria

The absolute floor comes first, and it is the least interesting. The candidate must exceed a minimum on the primary metric, and the minimum should derive from what the business needs rather than from what previous models achieved, or it ratchets downwards over time. A floor also protects against the degenerate case where both models are bad and the challenger is marginally less bad.

The comparison against the incumbent is the substantive test, and the subtlety is that it must clear noise rather than zero. If your seed-to-seed variance on the headline metric spans half a point, a candidate that beats the incumbent by a tenth of a point has beaten nothing, and promoting it is coin-flipping with extra ceremony. Measure that variance by rerunning your own configuration across seeds so the margin required is empirical. On a large evaluation set a confidence interval or a paired significance test on per-example differences serves the same purpose and is stronger, because the two models are being compared on identical examples.

The regression and fairness checks are where a mature gate differs from a naive one. An aggregate improvement can and routinely does conceal a subgroup that got worse, because aggregate metrics are weighted by population size and the interesting segments are small. So the gate evaluates the metric per slice — geography, channel, customer tenure band, device type, product line, and any protected attribute your jurisdiction obliges you to examine — and fails on a material regression in any slice even when the total improved. Where fairness is a regulatory requirement rather than a preference, the specific parity measure is not yours to choose freely and should be the one your compliance function has committed to. Alongside these sit behavioural regression tests: a set of held-out cases with known correct handling, including the edge cases that caused past incidents, which the model must still get right. This is the closest analogue to a unit-test suite that a model has, and it is how you stop a fixed bug from returning.

The operational criteria are gated too, because a model that is more accurate and twice as slow may be a net loss. Inference latency at the percentile in your service objective, measured on production-like hardware. Artefact size, since it affects cold-start time. Memory footprint. Prediction distribution and calibration compared against the incumbent, because a large shift in the score distribution will break every downstream threshold even if ranking quality improved. And the artefact must load and serve at all — a smoke test that scores one request through the real serving stack, which catches the serialisation and dependency problems that no metric will.

flowchart TD
  A[Candidate registered] --> B[Score candidate<br/>and incumbent together]
  B --> C{Absolute floor<br/>and margin over noise}
  C -->|fail| X[Reject<br/>record report]
  C -->|pass| D{Per-slice and<br/>fairness regression}
  D -->|fail| X
  D -->|pass| E{Latency, size,<br/>calibration shift}
  E -->|fail| X
  E -->|pass| F[Approve<br/>human step if regulated]

The property worth noticing is that a rejection still writes a report to the registry. A gate that silently discards failing candidates loses the record of what was tried, which is exactly the record you want when the fifth candidate in a row fails the same slice.

The evaluation set is a resource you can exhaust

A gate that runs on every retrain, comparing every candidate against a fixed evaluation set, is a selection process — and selection against a fixed holdout is overfitting, however clean the code is. Run twenty candidates against the same set and the winner is partly a winner because of that set's idiosyncrasies. Nothing in the gate detects this.

The mitigations are procedural rather than technical. Roll the evaluation window forward with time so successive candidates face genuinely new data, which the temporal holdout gives you for free provided labels are available. Keep a smaller set that is not used for routine gating and is only consulted at intervals, and treat a divergence between the two as a warning that the routine set has been mined. And be suspicious of a long sequence of candidates each improving by a hair — that pattern is more often holdout mining than steady progress.

Why the human step survives

The temptation, having built all of the above, is to conclude that a sufficiently thorough automated gate makes approval a rubber stamp. In consequential settings it does not, for three reasons that are worth articulating because they are not about distrust of automation.

The first is that a gate can only test what someone thought to encode. It cannot notice that the training data covered a period containing an unusual event, that a feature has become unavailable in a market you are about to enter, or that the business changed its policy last week in a way that makes the target definition stale. Those are judgements about context that live outside the evaluation set.

The second is accountability. In regulated model risk management the requirement is often that a named, suitably independent person is answerable for the model in production. That is not a statement about whether an algorithm could decide correctly; it is a statement about who answers to the regulator, and no automated gate can hold that position.

The third is that the approval step is a natural place to force documentation into existence. The reviewer needs the evaluation report, the lineage, the known limitations and the intended use, so requiring the review guarantees those artefacts are produced while the context is fresh rather than reconstructed under pressure later.

The design that reconciles both instincts is to let automation do all the work and reserve for the human only the decision. The gate computes everything, assembles the report, and stops at a transition the reviewer must effect. They are not recomputing metrics; they are accepting or refusing on the strength of a complete package, which takes minutes when the gate has done its job. Where the consequences are low — a recommendation ranker, an internal forecast — the same machinery can promote automatically, and the difference between the two paths should be a policy attached to the model's risk tier rather than a difference in the pipeline.

Compare challenger and incumbent on identical data in one run, require the margin to clear your measured seed noise, and check slices before believing an aggregate gain. Whether a human signs the transition is a question about consequence, not about how good your automation is.