Machine learning fundamentals

When the rule is already known, when there is no label, or when nobody will act on the output. A deterministic policy — block the payment if the country is on a sanctions list — should be an if statement, because a model would learn the rule imperfectly and make it unauditable. A problem with no recorded outcome is not a supervised problem at all until somebody starts recording it, and pretending otherwise produces a model fitted to a proxy nobody agreed to. And a prediction that changes no decision is a report, not a model, however accurate it is. The diagnostic question is what happens differently on the day the model is right; if the answer is "we look at a dashboard", the model was never the work.

A label you can obtain at scale and trust, the same features at prediction time that you have in training, and a decision the prediction changes. Each one kills projects independently. The label is the most common failure: it arrives months late, or it is produced by inconsistent human judgement, so the model learns the annotators rather than the phenomenon. Prediction-time availability is the quietest failure, because the warehouse contains fields that were written after the event you are predicting and nothing in the schema says so. And the decision determines everything downstream — the metric, the threshold, the latency budget, the acceptable error direction. Establishing all three takes a conversation, not a notebook, and doing it last is how six months disappear.

The model is only the middle of the loop, and every arrow in it is a place the two halves can disagree.

flowchart LR
    H[Historical data<br/>as of prediction time] --> F[Feature definition<br/>one implementation]
    F --> T[Training set<br/>with labels]
    T --> M[Fitted model<br/>plus a threshold]
    F --> S[Serving feature lookup]
    S --> P[Scored request]
    M --> P
    P --> O[Outcome recorded<br/>becomes a later label]

The single feature definition feeding both paths is the load-bearing part of this diagram. If training computes a thirty-day average in warehouse SQL and serving computes it in application code, you have two features with one name, and the model is being asked questions in a dialect it was not taught. Nothing alerts, because offline evaluation uses the offline definition.

as of prediction time on the first node is the other constraint that gets dropped. A training row dated June may only use data that existed in June, which means the join is point-in-time rather than a straightforward equality on customer identifier. Most leakage is a violation of that one arrow.

The threshold travels with the model, not with the code that calls it. A classifier emits a score; the number that turns a score into an action is a business decision, and versioning it separately from the model is how a retrain silently changes the alert volume the operations team receives.

The loop closing at the bottom is what makes the system improvable and what makes it dangerous. Outcomes recorded on scored requests become the next training set, so the model's own behaviour shapes the data it will next learn from — you only observe outcomes for the cases you acted on, and the cases you rejected have no label at all.

To establish what the number you report actually means, because a metric with no reference point is unreadable. Three baselines are worth having. The trivial one — predict the majority class, or predict last week's value — tells you whether the model has learned anything at all, and on imbalanced or strongly autocorrelated data it is embarrassingly strong. The existing process, whether that is a rules engine or a human team, is the one the business compares against, and beating a model that does not exist is not a result. And a simple model such as regularised logistic regression tells you how much of the achievable signal is linear, which sets your expectation for how much a complicated model can add. Reporting the baseline unprompted is a reliable competence signal.

By working backwards from the action, then checking which candidate target you can actually observe. If the action is offering a retention discount, "will churn within ninety days" is a better target than "is dissatisfied", because the first is a recorded event with a date and the second is an unmeasurable state. Then check the horizon against how long the intervention needs: predicting churn one day out is accurate and useless, because there is no time to act. Then check the base rate, since a target so rare that a month of data holds forty positives will not support a model regardless of technique. The target that survives all three is often narrower and more boring than the one the business asked for, and saying so is the job.

It sets the clock on everything. If a loan default is only observable after ninety days, then your freshest usable training data is ninety days old, your first honest evaluation of a newly shipped model is ninety days away, and any concept drift is discovered a quarter after it started. That forces a two-tier monitoring design: feature and prediction distributions watched immediately as an early warning, true performance measured against ground truth when it arrives. It also argues for a short-horizon proxy target you can measure weekly, used strictly for drift detection rather than for the decision. The cost people miss is experimental: with a ninety-day label, you get four honest model comparisons a year, so the sequence in which you try things matters far more than usual.

Because it decides what a row is, and therefore what the metric measures and what the model can possibly learn. Predicting "will this machine fail" per machine per month, per machine per day, and per component per day are three different problems with different base rates, different feature windows and different usable actions. Get it wrong and every number afterwards is coherent and meaningless: a per-day model evaluated per month will look far better than it is, because one correct day out of thirty counts as a caught failure. The unit also fixes the independence assumption behind your evaluation — many rows per machine means rows are correlated, and a random split will put the same machine on both sides. Declare the unit before choosing anything else.

Task: predict delivery time in minutes. Metric: mean absolute error.

model                             train MAE   validation MAE
--------------------------------  ---------   --------------
depth-2 decision tree                  8.4              8.6
depth-12 decision tree                 1.1              7.9
boosted trees, early stop at 220       4.2              5.3

Irreducible floor: two dispatchers estimating the same completed
delivery from the same information disagree by MAE 3.1, so no model
reading that information can do better than about 3.1.

Read the gap between the two columns and the level of the second one, and the diagnosis falls out. The depth-2 tree is high bias: it is equally wrong on data it has seen and data it has not, which means the model cannot express the pattern and more data will not help. Adding rows to an underfitting model is the most common wasted quarter in applied machine learning.

The depth-12 tree is high variance: near-perfect on training and barely better than the stump on validation, so it has memorised the sample rather than learned the signal. Here more data genuinely does help, as do the constraints — shallower trees, fewer features, stronger regularisation.

The boosted model is 2.2 above the floor with a 1.1 train-validation gap, so there is real but limited headroom, and most of it is in features rather than in the model. That is the reading that changes what you do next.

The floor is the part candidates omit and it reframes the whole exercise. A stated irreducible error turns "the model is only 5.3" into "the model is within 2.2 of what the information permits", and it stops the team from chasing an accuracy the data cannot contain. Deriving it from annotator disagreement, or from the variance of repeated measurements, is worth the day it takes.

When the relationship is close to linear, you need coefficients someone will read, and you want an answer today. Its virtues are all consequences of its simplicity: it extrapolates predictably beyond the training range, it trains on data that fits in a spreadsheet, and each coefficient is a statement in the units of the problem that a domain expert can confirm or reject. That last property is why it survives in regulated settings where a model must be explained to somebody who can refuse it. Its costs are exactly as visible: it cannot represent an interaction or a threshold effect unless you construct the term yourself, it is dragged around by outliers because the squared loss punishes them quadratically, and correlated predictors make its coefficients unstable and therefore uninterpretable.

Because it produces a well-behaved, roughly calibrated probability from a convex optimisation that has no hyperparameters worth agonising over beyond the regularisation strength. That makes it the honest baseline: it trains in seconds, it tells you how much of the signal is linear in your features, and its coefficients let a domain expert spot a leaking feature immediately, which a gradient-boosted model will not. The probability being usable straight out is the underrated part, since it means you can plug it into an expected-value calculation without a calibration step. The cost is the same as for linear regression — interactions and non-monotone effects have to be engineered by hand — plus the fact that on genuinely non-linear tabular problems it will lose to boosted trees by a margin worth having.

Split on thresholds and therefore represent interactions and non-monotone effects without being told they exist. A tree can learn that risk rises sharply above a particular loan-to-value ratio only for a particular product, which a linear model can express only if you construct that interaction term yourself. It is also indifferent to feature scale and to monotone transformations, since it only ever compares a value against a split point, and it handles mixed types without one-hot encoding everything. The costs are severe on its own: a single tree is high variance, so re-fitting on a slightly different sample gives a visibly different tree, and it cannot extrapolate beyond the range it saw — every prediction is an average of training rows in some leaf. Those two costs are what ensembles exist to fix.

Bagging fits many models independently on bootstrap samples and averages them, which attacks variance: the individual trees are deliberately deep and overfitted, and averaging cancels the parts of their error that are uncorrelated. It parallelises perfectly and is almost impossible to overfit by adding more trees. Boosting fits models sequentially, each one on the errors the ensemble so far still makes, which attacks bias: the individual learners are deliberately weak, and the ensemble becomes expressive by accumulation. That sequence means boosting cannot be parallelised across trees, and it can and will overfit if you keep adding them, so it needs early stopping against a validation set. The practical consequence is that bagging is forgiving and boosting is stronger when tuned.

Because tabular data has the properties trees are built for and lacks the properties neural networks exploit. The columns are heterogeneous — a currency amount beside a category code beside a count — so there is no spatial or sequential structure for convolution or attention to share weights across. Many features are irrelevant, and a tree simply never splits on them, whereas a dense layer must learn to zero them out. Relationships are often sharp thresholds rather than smooth functions, which is precisely what a split represents and what a smooth activation approximates awkwardly. Boosting then adds the bias reduction, so the ensemble fits residual structure that a single tree misses. The cost is real: you get no useful probability without calibration, no gradient to fine-tune, and an ensemble of several hundred trees to explain.

Six predictors, where x1 and x2 are near-duplicates and x4 and x5 are noise, fitted three ways on the same data.

                      unpenalised      L2 ridge       L1 lasso
x1  near-duplicate           8.41           1.52           2.68
x2  near-duplicate          -5.63           1.31           0.00
x3  real signal              1.90           1.44           1.61
x4  noise                    0.31           0.09           0.00
x5  noise                   -0.22          -0.06           0.00
x6  real signal              0.88           0.71           0.77

sum of |coefficients|       17.35           5.13           5.06
non-zero coefficients            6              6              4

The unpenalised fit shows what collinearity does: x1 and x2 carry nearly the same information, so the loss is almost flat along the ridge where one rises and the other falls, and the solver lands somewhere arbitrary on it — here with opposite signs, which reads as "x2 reduces the outcome" and is a statement about the sample rather than the world. Re-fit on a different 90% of the rows and both numbers move substantially.

L2 penalises the sum of squared coefficients, so it is cheapest to split a shared effect between correlated predictors rather than load it onto one. Both keep a moderate positive weight, the noise terms shrink towards zero without reaching it, and the whole solution becomes stable across resamples. Nothing is selected out, so you still have six features to serve and explain.

L1 penalises the sum of absolute values, whose gradient does not vanish as a coefficient approaches zero, so the optimum frequently sits exactly at zero. It picks one of the duplicated pair and discards the other, and it drops both noise terms — genuine feature selection as a side effect of fitting.

The costs follow directly. L1's choice between x1 and x2 is arbitrary and will flip on a resample, so the selected set is unstable and reading it as "x2 does not matter" is wrong. L2 keeps everything, so it does nothing for serving cost. The elastic net exists because most real problems want both behaviours, and the penalty strength is a hyperparameter chosen on validation data rather than an opinion.

When something other than the metric is binding, which is most of the time in production. A logistic regression that scores two points lower may be the right choice when a regulator requires an explanation of each decision, when the serving budget is two milliseconds and the boosted ensemble takes twenty, when the team that will own it at three in the morning cannot debug a hundred-megabyte artefact, or when the feature the strong model depends on comes from a pipeline that fails weekly. The cost of the better model is rarely paid in accuracy; it is paid in latency, explainability, retraining fragility and operational surface. Naming which constraint you are optimising against, and stating the accuracy you gave up to satisfy it, is the difference between a considered choice and an unexamined one.

Trees learn interactions between features you gave them; they cannot construct a quantity that is not derivable by axis-aligned splits. A ratio is the clearest case — a tree can approximate "amount divided by the customer's usual amount" only by carving the plane into rectangles, needing many splits and many rows to do badly what one division does exactly. The same applies to aggregations over a window, differences from a rolling median, counts within a time horizon, and anything requiring a join. That is where most real gains live, and it is why the strongest tabular models are usually a modest algorithm over thoughtful features rather than the reverse. The cost is that every engineered feature is a definition that must be reproduced identically at serving time, so each one adds a place the two paths can diverge.

By deciding what you want the encoding to carry. One-hot is safe and honest below perhaps fifty levels, and above that it produces a wide sparse matrix in which rare levels have too few rows to support a coefficient. Grouping the tail into a single other bucket, chosen by a frequency threshold fixed on training data alone, handles most of the problem and is boringly robust. Frequency encoding replaces the level with its count, which is useful when rarity itself is predictive. Target encoding replaces it with the mean outcome for that level, which is powerful and leaks unless it is computed out of fold and smoothed. Learned embeddings pay off only when cardinality is very large and you have the rows to fit them. The unseen level at serving time needs an explicit policy in every one of these.

The encoding is fitted on the whole training set, and the rare levels do the damage.

merchant   rows   frauds   naive mean encoding
--------   ----   ------   -------------------
M-0009     8400      260                0.0310
M-1120      940       35                0.0372
M-4417        1        1                1.0000   <- this row's own label
M-5581        2        0                0.0000   <- these rows' own labels

Global fraud rate = 0.0120.

Smoothed, out of fold, with a prior weight m = 20:
  encoded = (n x category_mean + m x global_mean) / (n + m)

  M-0009   (8400 x 0.0310 + 20 x 0.0120) / 8420 = 260.4 + 0.24 / 8420 = 0.0310
  M-1120   ( 940 x 0.0372 + 20 x 0.0120) /  960 =  34.97 + 0.24 / 960 = 0.0367
  M-4417   (   1 x 1.0000 + 20 x 0.0120) /   21 =   1.0 + 0.24 /   21 = 0.0590
  M-5581   (   2 x 0.0000 + 20 x 0.0120) /   22 =   0.0 + 0.24 /   22 = 0.0109

The naive column for M-4417 is the label, copied into a feature under a different name. Any model will learn to trust that feature completely, validation will look superb, and in production the merchant is either unseen or has an encoding computed from history that carries none of that signal. The score collapses and nothing in the code looks wrong.

Smoothing fixes the magnitude problem. A level with 8,400 rows keeps its own mean essentially unchanged, because the prior's weight of 20 is negligible beside it, while a level with one row is pulled to 0.059 — still above the global rate, which is honest, and no longer a restatement of the answer. The prior weight is the single knob, and it is chosen on validation data rather than asserted.

Out of fold is the other half and it is not optional. Each row's encoding must be computed from the other folds only, so a row never contributes to its own feature value, and the final encoding applied at serving time is fitted on the full training set. Doing the smoothing without the fold discipline still leaks for mid-sized levels.

For anything with a time dimension, folds are not enough either: the encoding must be computed from data strictly before the row's timestamp, or a merchant's future fraud is informing its past. That is the version that survives contact with production.

Anything whose loss or distance depends on the magnitude of a feature. Gradient descent converges badly on unscaled inputs because the loss surface becomes a long narrow valley, so neural networks and linear models fitted iteratively need it. Regularised linear models need it for a subtler reason: the penalty is applied to coefficients, and a feature measured in pounds gets a smaller coefficient than the same feature measured in thousands of pounds, so the penalty silently discriminates by unit. Distance-based methods — k-means, k-nearest-neighbours, anything using Euclidean distance — are dominated by whichever feature has the largest range. Trees and tree ensembles need none of it, since a split only ever asks whether a value exceeds a threshold. The rule that matters more than the list: the scaler is fitted on training data and applied elsewhere, never fitted on everything.

The feature table joins cleanly, the column names are innocent, and every value was recorded after the moment you are predicting.

Predicting: will this loan application default within 12 months?
Prediction is made at applied_at. Label observed 12 months later.

feature                       populated when          usable at applied_at?
---------------------------   ---------------------   ---------------------
applicant_income              at application          yes
existing_balance              at application          yes
n_collections_calls           when collections chase   NO - only after default
account_status_code           updated continuously     NO - reads "closed" now
employer_verified_flag        during manual review     NO - review happens after
postcode_risk_band            rebuilt monthly from     NO - the band was fitted
                              observed defaults            on these very defaults

One row, joined as of today rather than as of applied_at:
  applied_at        2025-03-11
  n_collections_calls        7   <- calls made in Nov 2025
  account_status_code  CLOSED    <- closed Jan 2026 after default
  label                 default

Three of these six features are the label wearing a different name. n_collections_calls is non-zero almost exclusively for accounts that defaulted, so the model discovers that seven calls means default, which is true and useless: at application time the value is always zero for everybody.

account_status_code is the most dangerous shape, because the column is legitimate and only the join is wrong. The warehouse holds the current status, and joining on account identifier gives you the status as of today rather than as of the application. That is a point-in-time correctness bug, and the fix is a history table with validity ranges plus a join predicate that bounds valid_from and valid_to around applied_at.

postcode_risk_band leaks through an artefact rather than a row. It was fitted on observed defaults including the ones in your training set, so it carries aggregate label information. Any derived score, segment or band built elsewhere in the business must be interrogated for what it was fitted on and when.

The detection habits that actually work: rank feature importances and treat a single dominant feature as a leakage hypothesis before celebrating it; ask for each feature the literal question of what its value was at the prediction timestamp; and split by time so that a feature depending on the future degrades visibly instead of scoring well. A validation number that is surprisingly good is evidence of a bug, not of skill.

By first asking why the value is missing, because that determines whether an imputation is defensible. If a field is missing because the customer is new, the missingness is itself the strongest feature and imputing the median destroys it — add an explicit indicator column and let the model use it. If it is missing because an upstream pipeline failed, imputation is papering over a data incident you should be alerted to. Median or mode imputation is a reasonable default for genuinely random gaps, fitted on training data only; model-based imputation adds complexity and a second model to maintain for usually modest gain. Some tree implementations learn a default direction for missing values natively, which is often better than anything you would construct. The cost of any imputation is that you have invented data and the model cannot tell.

Any field whose value at prediction time differs from its value in your training snapshot, any field that will not exist in the serving path, and any field you are not legally permitted to decide on. The first is the point-in-time problem: a mutable column joined as of today is a different feature from the same column joined as of the event. The second is mundane and lethal — a warehouse aggregate computed nightly cannot be read inside a fifty-millisecond scoring call unless somebody builds that path. The third is protected characteristics and their close proxies, where a postcode can encode ethnicity well enough to constitute indirect discrimination whether or not you intended it. The discipline is to justify each feature's availability and legitimacy before its predictive power, because a feature that fails either test cannot be rescued by being useful.

Because it assumes rows are independent, and in real datasets they usually are not. If a patient, customer or machine contributes many rows, a random split puts near-duplicates of a training row into the test set, so the model is being graded on data it has effectively seen and the score measures memorisation of entities rather than generalisation to new ones. If the data has a time dimension, a random split trains on the future and predicts the past, which is a capability you will not have in production. Both inflate the estimate in the same direction and by an amount you cannot bound from the score alone. The fix is to make the split match the generalisation you actually need — unseen groups, or unseen future — and to expect the honest number to be materially worse.

Five folds over eighteen months of data, where the same machine appears many times and predictions are made two weeks ahead.

Wrong: KFold shuffle=True over rows
  machine 4417 has 540 rows spread across all five folds
  the model memorises machine 4417 and is graded on machine 4417

Better: GroupKFold on machine, still ignoring time
  each machine sits in exactly one fold - generalises to new machines
  but fold 2 may train on August and test on May

Right for a forecast: expanding window, purged, grouped where needed

  fold  train window            gap      test window
  ----  ---------------------   ------   ---------------------
   1    2025-01 .. 2025-06      2 wks    2025-07 .. 2025-08
   2    2025-01 .. 2025-08      2 wks    2025-09 .. 2025-10
   3    2025-01 .. 2025-10      2 wks    2025-11 .. 2025-12
   4    2025-01 .. 2025-12      2 wks    2026-01 .. 2026-02
   5    2025-01 .. 2026-02      2 wks    2026-03 .. 2026-04

  Train always ends before test begins. Train grows; test is a fixed
  two months. Five estimates, each on genuinely unseen future.

The gap is the detail almost nobody includes and it is not decoration. With a two-week prediction horizon, a training row dated the last day of June has a label determined by events into mid-July, which overlaps the test window — so without the gap the training labels contain test-period information. Purging by at least the horizon removes that overlap.

The expanding window is chosen over a sliding one when older data still carries signal and volume is the binding constraint. A sliding window of fixed length is the right choice when the process changes enough that two-year-old behaviour is actively misleading, and which one to use is an empirical question you settle by running both.

Read the variance across the five folds as a result in its own right. If fold 1 scores well and fold 5 badly, that is not noise to average away — it is drift, and the number you should quote to the business is the most recent fold, because that is the regime the model will run in.

The combination matters when both structures are present. Split by time first, then ensure no group straddles the boundary, because a machine appearing in both training and test at different dates still leaks entity-specific quirks.

Two phishing detectors on the same corpus, compared at the same recall, with both denominators written out.

100,000 emails, 500 phishing -> base rate 0.5%, negatives 99,500

Both models held at recall 0.80, so both catch 400 of 500.

              false      false positive rate      precision
              positives  FP / 99,500              TP / (TP + FP)
------------  ---------  -----------------------  -----------------
model A           1,600  1,600 / 99,500 = 0.016   400 / 2,000 = 0.200
model B           6,400  6,400 / 99,500 = 0.064   400 / 6,800 = 0.059

Random baseline:  ROC-AUC 0.50          PR-AUC ~ 0.005 (the base rate)

The two metrics disagree because they divide by different things. The false positive rate divides by the 99,500 negatives, so model B's four extra thousand false positives move it from 1.6% to 6.4% — both of which read as "fine" and sit in the top-left corner of an ROC curve. Precision divides by the model's own positive predictions, so the same difference shows up as 20% against 5.9%.

Translate that into the thing a business feels. At model A, a reviewer opens five flagged emails to find one attack; at model B, seventeen. That is a 3.4-fold difference in headcount for identical detection, and ROC-AUC will report the two models as nearly equivalent.

The baselines complete the argument. A random model scores 0.5 on ROC-AUC, so 0.96 sounds like most of the way to perfect; on PR-AUC a random model scores about the prevalence, 0.005, so any improvement is legible against a floor that reflects how hard the problem is. PR-AUC also moves when the base rate moves, which is a feature and not a flaw: it means the metric changes when the problem changes.

Use ROC-AUC when the classes are roughly balanced and both error types cost comparably, or when you want a metric that is invariant to prevalence for comparing across populations. Use PR-AUC, and report precision at your actual operating point beside it, whenever positives are rare and somebody has to act on each flag.

Which threshold you will run, what it costs there, or whether the probabilities mean anything. AUC is the probability that a random positive scores above a random negative, so it is a summary of ranking quality averaged over every possible threshold — including the vast majority you would never operate at. Two models with identical AUC can differ enormously in the region you care about, one being excellent at the very top of the ranking and the other making up its score in the middle. It is also entirely insensitive to calibration: apply any monotone transformation to the scores and the AUC is unchanged while every probability becomes wrong. So AUC is a reasonable model-selection signal during development and a poor thing to report to the business, which needs precision and recall at the operating point.

Because the training objective rewards ranking and loss reduction, not truthfulness of the score's magnitude, and several standard practices push it further out. Boosting with many trees drives scores towards the extremes, so predictions cluster near zero and one and moderate uncertainty is under-represented. Bagged ensembles do the opposite and pull scores towards the mean, since averaging many trees rarely produces a unanimous value. Any resampling or class weighting used to handle imbalance shifts the whole score distribution away from the true base rate by construction. The practical consequence is that using such a score directly in an expected-value calculation — multiply by the loss, compare against the cost of acting — gives an answer that is wrong by whatever the miscalibration is, while the ranking and therefore the AUC look perfect.

Bucket the predictions, compare the mean predicted probability against the observed frequency in each bucket, and total both columns.

Churn model, 10,000 customers scored, outcomes now known.

bucket      n      mean predicted   observed rate      predicted   actual
                   probability      churned / n        churners    churners
---------  -----   --------------   ---------------   ---------   --------
0.0-0.2    6,000            0.08     300 / 6,000 = 0.050     480       300
0.2-0.4    2,000            0.29     440 / 2,000 = 0.220     580       440
0.4-0.6    1,200            0.49     504 / 1,200 = 0.420     588       504
0.6-0.8      600            0.69     366 /   600 = 0.610     414       366
0.8-1.0      200            0.88     151 /   200 = 0.755     176       151
---------  -----                                          -------   -------
total     10,000                                            2,238     1,761

Observed rates are strictly increasing, so the ranking is perfect and
ROC-AUC is unaffected. Predicted total exceeds actual by 477, which is
2,238 / 1,761 = 1.27, so every probability is about 27% too high.

The two failures this table separates are discrimination and calibration. Discrimination is fine here: the observed rate rises monotonically through every bucket, which is exactly what AUC measures, so a model-selection process driven by AUC would have found nothing wrong. Calibration is broken: each bucket's predicted value sits above its observed rate, consistently.

That gap becomes money the moment the score enters a decision. If retention offers are sent where predicted churn times customer value exceeds the offer cost, a 27% overstatement means offers going to customers who were never going to leave, and the campaign's forecast return is wrong before it starts. Nothing in the offline evaluation surfaces this, because the evaluation graded the ranking.

The fixes are both post-hoc and both fitted on held-out data the model did not train on. Platt scaling fits a logistic regression from score to outcome, which is parametric and works well when the distortion is a smooth sigmoid. Isotonic regression fits any monotone mapping, which is more flexible and needs more data because it can overfit a small calibration set. Either leaves the ranking untouched by construction, so AUC is unchanged and the probabilities become usable.

The check has to be re-run after every retrain and on production data rather than the original test set, because calibration decays with base-rate drift while discrimination often survives it. A model whose base rate moved is miscalibrated before it is inaccurate.

RMSE squares the errors, so it is dominated by the largest ones and is the right choice when a single big miss is disproportionately costly — capacity planning, where being ten units short once matters more than being one unit short ten times. MAE treats every unit of error alike, which makes it robust to outliers and easier to explain to a stakeholder as "we are typically wrong by four minutes". MAPE expresses error relative to the actual, which lets you compare across series of different sizes, and it blows up as the actual approaches zero and asymmetrically punishes over-prediction, so it is a poor choice for intermittent demand. The consequential point is that the metric encodes a loss function you are asserting on the business's behalf, and it should be the one they would choose if asked.

Probably not, and the way to find out is to look at variance rather than at the point estimate. Refit both models across the cross-validation folds and compare the distributions: if the fold-to-fold standard deviation is 0.01, a 0.004 difference is inside the noise and reporting it as an improvement is reporting a seed. Re-run with several random seeds too, because for anything with stochastic initialisation or subsampling the seed alone moves the metric by a comparable amount. Then ask the question that usually settles it: does the difference change any decision at the operating point you will run? A gain that leaves precision at your alert budget unchanged is not an improvement in anything the business will notice, whatever the third decimal place says.

It makes accuracy a measure of how often the model says no, since predicting the majority class for everything scores 99.8%. It makes the false positive rate nearly unreadable, because its denominator is the enormous negative class, so a model producing ten false alarms for every true one still shows a fraction of a per cent. It makes precision fragile: at 0.2% prevalence even a very specific model flags mostly negatives, which is arithmetic rather than a modelling failure. It makes your effective sample size the positive count, not the row count, so two million rows with four thousand positives is a small dataset. And it makes cross-validation noisy, because a fold may contain few enough positives that the estimate swings on individual cases — which is what stratification exists to mitigate.

Class weights multiply the loss contribution of minority errors, which uses all the data, requires no resampling machinery, and is a one-line change in most libraries; the cost is that the resulting scores are no longer calibrated to the true base rate. Under-sampling the majority trains quickly and discards information, which is defensible when the majority class is enormously redundant and wasteful otherwise. SMOTE interpolates new minority points between existing neighbours, which can help a distance-based or linear model see a broader minority region, and it fabricates data — in high dimensions or with categorical features the interpolated points may be implausible, and if applied before the split it leaks by placing synthetic relatives of test rows into training. All three change the score distribution, so recalibrate if you need probabilities.

The model is fixed; only the cut-off moves. The costs are stated assumptions, and every row is arithmetic from them.

20,000 machine-days scored per month, 200 true failures -> 1.0% positive.
Missed failure: unplanned outage, £6,000.  Inspection: £150.
Doing nothing at all: 200 x £6,000 = £1,200,000.

thresh   TP    FN    FP     inspections   FN cost    inspect cost   total
------  ----  ----  -----   -----------   ---------  ------------  ----------
 0.10    180    20  2,400         2,580   £120,000      £387,000    £507,000
 0.30    150    50    700           850   £300,000      £127,500    £427,500
 0.50    120    80    240           360   £480,000       £54,000    £534,000
 0.70     80   120     60           140   £720,000       £21,000    £741,000

precision and F1 at each threshold:
 0.10   p = 180/2,580 = 0.070   r = 0.90   F1 = 0.130
 0.30   p = 150/  850 = 0.176   r = 0.75   F1 = 0.285
 0.50   p = 120/  360 = 0.333   r = 0.60   F1 = 0.426
 0.70   p =  80/  140 = 0.571   r = 0.40   F1 = 0.471

The cost column and the F1 column point in opposite directions. Cost is minimised at 0.30, where precision is 17.6% — five inspections per genuine fault, which reads like a bad model. F1 is maximised at 0.70, which costs £741,000, or £313,500 more than the cost-optimal choice. Selecting a threshold by F1 here is a £313,500 decision made by a metric that was never told what anything costs.

The reason is structural rather than accidental. F1 is the harmonic mean of precision and recall, which weights them equally, and this problem weights them at forty to one — a missed failure costs £6,000 against £150 for an inspection. Any single metric that does not contain the cost ratio will pick the wrong point whenever that ratio is far from one.

The no-model baseline belongs in the table. £1,200,000 of avoidable outage is what the business bears today, so the 0.30 threshold saves £772,500 a month, and that figure is what justifies the project. A precision of 17.6% quoted without it sounds like failure.

Two refinements worth volunteering. Inspection capacity is usually a hard constraint, so if the team can only handle 400 inspections a month the 0.30 row is unavailable regardless of its cost and you are choosing among feasible thresholds — which is the fixed-budget framing. And the cost per inspection is not constant: send too many and the team starts skipping them, so the real curve bends in a way no static table shows and only an operating review will surface.

The constraint is not the metric, it is that a human team can only look at so many cases a day, and that changes which model you pick.

200,000 transactions a day, 400 fraudulent -> 0.2% base rate.
The fraud operations team can review 500 cases a day. That is the budget.

Rank every transaction by score, take the top 500.

model   ROC-AUC   frauds in top 500   precision@500       recall@500
-----   -------   -----------------   -----------------   -----------------
  A       0.970                 120   120/500 = 0.240     120/400 = 0.300
  B       0.952                 160   160/500 = 0.320     160/400 = 0.400

Model A wins on AUC by 0.018 and loses 40 frauds a day.
At £180 average loss per missed fraud: 40 x £180 = £7,200 a day,
which is £7,200 x 365 = £2,628,000 a year.

Model A is genuinely the better ranker overall and the worse model to deploy. AUC averages ranking quality across the entire score range, and the budget means you only ever operate in the top 0.25% of it, so A's superiority in the middle of the distribution is bought with weakness exactly where you look.

That is the general lesson: evaluate at the operating point the deployment imposes, not on a summary of all operating points. When capacity is fixed, the metric is precision at k or recall at k with k set to the real budget, and it should be the number in the model-selection table from the first experiment onwards.

The budget also reframes what an improvement is. Doubling the team's capacity to 1,000 moves you further down the ranking where precision is lower, so the marginal fraud caught per reviewer falls — there is a diminishing return curve you can compute from the same ranking, and it tells the business what a headcount request actually buys.

The trap is that recall@500 is capped by the budget, not by the model: with 400 frauds and 500 slots, perfect recall is arithmetically possible here but would demand a near-perfect ranker. On a day with 800 frauds, recall@500 cannot exceed 0.625 no matter how good the model is, and reporting a recall decline on that day as model degradation is a mistake.

Because you changed the base rate the model was fitted on, and the model's output is an estimate of the probability in the data it saw. Balance a 1% positive class by under-sampling the majority to 50/50 and the model's scores now describe a world in which half of all events are positive, so a predicted 0.4 corresponds to something far rarer in reality. Ranking is largely preserved, so AUC looks fine and nothing appears broken until the score is used in an expected-value calculation or shown to a user as a percentage. The remedies are to avoid resampling and use class weights with a threshold chosen on the real distribution, or to correct the scores analytically for the sampling ratio, or to recalibrate on a held-out sample with the true prevalence intact.

That somebody has specified recall and left precision unbounded, which fixes the easy half of the problem and hides the hard half. Recall of 95% is always achievable — flag everything — so the requirement is empty until you attach the precision or volume you will tolerate at that recall. The useful response is to convert it into the operational quantity: at 95% recall this model produces so many alerts a day, which needs so many reviewers, and here is the recall you can have for the team you actually have. Then let the business choose the point. Doing this early also flushes out whether 95% was a considered figure or a round number, and it usually turns out that the real constraint was the alert volume all along.

When the positive count, rather than the row count, is too small to support estimation and validation. A few dozen positives cannot be split into training and test folds that give a stable estimate, and any model fitted on them will be learning the idiosyncrasies of individual cases. The honest options at that point are to widen the target so positives accumulate — predict "any safety incident" rather than one specific failure mode — to lengthen the observation window, to aggregate the unit of prediction upwards, or to abandon supervised learning for unsupervised anomaly detection plus expert rules, which needs no positives at all. Saying that a problem does not yet have enough positives, and specifying what would change that, is a stronger answer than fitting a model to forty events and reporting its cross-validated score.

With internal measures, stability checks, and a downstream test — in increasing order of how much they are worth. Internal measures such as silhouette or the Davies-Bouldin index score compactness against separation, which tells you whether the geometry is clean and nothing about whether the grouping is meaningful. Stability is more informative: cluster repeated subsamples and measure how consistently pairs of points end up together, because a solution that reshuffles on 90% of the data is an artefact of the sample. The real evaluation is external and downstream — do the clusters differ on variables you did not cluster on, do domain experts recognise them, and does treating them differently change an outcome. A clustering nobody acts on cannot be evaluated, which is usually a sign it should not have been built.

Because its assumptions are strong and mostly unstated. It minimises within-cluster squared distance, which biases it towards spherical, similarly sized, similarly dense groups, so an elongated or crescent-shaped cluster gets cut in half and a small cluster next to a large one gets absorbed. It requires k in advance, and the elbow method that supposedly chooses it is frequently ambiguous on real data. It is driven by feature scale, so an unscaled column measured in thousands dictates the whole partition. It is sensitive to initialisation, which good seeding mitigates rather than removes, and it assigns every point to a cluster, so outliers are forced into one and drag its centre. The deeper problem is that it always returns k clusters, including when the data has no cluster structure at all.

The word covers three different problems, and a detector built for one is blind to the others.

shape        what it is                     example              suited to
----------   ----------------------------   ------------------   ----------------
point        a single record unlike all      one £40,000 card    isolation forest,
             the others in the dataset      payment on a         one-class SVM,
                                            £30-average account  autoencoder error

contextual   a record normal in general      50 logins at 03:00  per-segment or
             but abnormal in its context    for a shift worker   conditional model,
                                            versus an analyst    residual from a
                                                                 forecast

collective   each record normal, the        200 small transfers  sequence models,
             sequence or set is not        of £9 each in an      aggregate features
                                            hour                 over a window

Point anomalies are what off-the-shelf detectors find, and they are the least interesting class in most mature systems, because the obvious outliers were covered by a rule years ago. A detector that keeps rediscovering them produces alerts the operations team already knows about.

Contextual anomalies are where the value usually is, and they require you to supply the context explicitly. The mechanism is almost always the same: model the expected value given the context — the customer's own history, the hour, the segment — and score the residual rather than the raw value. Nothing in an unconditional detector can find them, since every individual value sits well inside the global distribution.

Collective anomalies need the unit of analysis to change. No single £9 transfer is unusual; the pattern is only visible once you aggregate over a window per account, so the feature engineering decides whether detection is possible at all. This is the class that structuring and coordinated-abuse cases fall into, and it is the one people try to solve with a better algorithm instead of a better window.

Naming which shape you are hunting before choosing a method is the point. It determines the features, the unit of prediction and the evaluation, and skipping it is how a project spends a quarter finding large numbers.

Rotating the feature space so that the axes are the orthogonal directions of greatest variance, then letting you keep the first few and discard the rest. That compresses correlated features into fewer numbers, decorrelates the inputs — which helps linear and distance-based methods — and gives a defensible way to reduce dimensionality before clustering. The costs are specific. Variance is not importance, so a low-variance direction carrying the signal can be discarded while a high-variance nuisance direction is kept. The components are linear combinations of every original feature, so interpretability is gone and you can no longer say which input drove a prediction. It assumes linear structure. And it must be fitted on training data alone, since fitting it on everything is a leak that looks harmless.

Whenever you read anything from it beyond "there appear to be groups". These methods optimise for preserving local neighbourhoods, so distances between clusters on the plot are not meaningful, cluster sizes on the plot do not reflect sizes or variances in the data, and the number of visible blobs depends heavily on the perplexity or neighbour count you chose. Different random seeds produce visibly different layouts of the same data. They will also manufacture apparent clusters from data that is genuinely uniform, which is the failure that causes real harm because the picture is persuasive and lands in a slide deck. Use them to generate hypotheses and then test those hypotheses on the original features, and never use the embedding coordinates as model inputs unless you have verified they help.

It builds many trees by picking a feature at random and a split value at random, recursively, and records how many splits it takes to isolate each point. Anomalies are isolated in fewer splits because they sit in sparse regions, so the average path length across the forest becomes the anomaly score. That inverts the usual approach — it models isolation directly rather than modelling normality and measuring deviation — which makes it fast, linear in the sample size, and untroubled by high dimensionality or by needing a distance metric. Its weaknesses follow from the random axis-aligned splits: it struggles with anomalies defined by a combination of features rather than by extremeness on any one, it finds only point anomalies unless you engineer contextual features yourself, and it produces a score whose scale is arbitrary, so the threshold is chosen from your alert budget rather than from the score.

An explicit notion of what neighbourhoods are dense, which lets it find clusters of arbitrary shape and label points as noise rather than forcing them into a cluster. DBSCAN and its variants need no cluster count, will happily return a crescent or a ring that k-means would bisect, and produce an outlier set as a natural by-product. Local outlier factor extends the same idea to scoring, judging a point against the density of its own neighbourhood rather than against the whole dataset, which is what makes it able to flag a point that is only locally unusual. The costs are the ones that limit their use: they depend on a distance metric, which degrades in high dimensions, they need a neighbourhood radius that is hard to choose and assumed roughly uniform across the data, and they scale worse than a tree method on large samples.

The gradient of the loss with respect to every parameter, by applying the chain rule backwards through the computation graph and reusing intermediate results. The forward pass computes and caches each layer's output; the backward pass starts from the loss, computes how much it changes with respect to the final layer's output, and propagates that sensitivity down layer by layer, multiplying by each layer's local derivative on the way. The insight worth stating is that it is dynamic programming rather than a new kind of calculus: computing each parameter's derivative independently would repeat the same sub-expressions enormously many times, and caching them makes the backward pass roughly the cost of the forward one. Two consequences follow — memory is dominated by those cached activations, and any operation in the graph must be differentiable for the chain to hold.

Per-layer gradient norms after one epoch of an eight-layer network with sigmoid activations, printed from the bottom of the stack upwards.

layer   mean gradient norm
-----   ------------------
   8              4.1e-02
   7              9.8e-03
   6              2.3e-03
   5              5.4e-04
   4              1.3e-04
   3              3.0e-05
   2              7.1e-06
   1              1.7e-06

Ratio between adjacent layers is roughly 0.24.

Why: the derivative of the sigmoid is at most 0.25, at its midpoint, and
smaller everywhere else. Seven layers of backward multiplication therefore
scale the gradient by at most 0.25^7 = 1/16,384 = 6.1e-05, and the observed
ratio 1.7e-06 / 4.1e-02 = 4.1e-05 is consistent with that bound.

At learning rate 1e-3, layer 1 sees updates of order 1.7e-09 against
weights of order 1e-01, a relative change of about 1e-08 per step. It is
frozen at initialisation.

The symptom in practice is not an error. Training loss falls, because the top layers are learning normally, and then plateaus well above where it should — the network is effectively a shallow model sitting on top of a random projection. Reading the loss curve alone gives you no way to see this, which is why logging per-layer gradient norms is the diagnostic worth having from the start.

The cause is structural rather than a bad choice of numbers. Backpropagation multiplies local derivatives along the path, and any activation whose derivative is bounded well below one makes that product shrink geometrically with depth. The mirror-image failure is a product of factors above one, which explodes instead, shows up as a loss going to not-a-number, and is treated with gradient clipping.

The fixes all attack the product rather than the layers. Activations whose derivative is one over the active region — the rectified family — stop the decay, at the cost of units that can die when their input stays negative. Residual connections give the gradient an additive path that skips the multiplications entirely, which is what made very deep networks trainable. Normalisation layers keep pre-activations in the range where derivatives are healthy. And careful initialisation scaled to layer width keeps the product near one at the start, which is where the trouble begins.

Adam fixes the problem that one global step size is wrong for parameters whose gradients differ in magnitude by orders of magnitude. It keeps running estimates of each parameter's gradient mean and squared magnitude, and divides the step by the latter, so rarely updated parameters take proportionally larger steps and noisy ones take smaller. That makes it forgiving about the initial learning rate, which is why it is the default. A schedule fixes something Adam does not: the step size should be small at the start while the parameters are random and the gradient estimates are unreliable, larger through the middle to make progress, and small at the end to settle rather than bounce around a minimum. Warmup and decay are the two halves, and skipping warmup is a common cause of early divergence.

It normalises each layer's pre-activations across the batch to roughly zero mean and unit variance, then applies a learned scale and shift. The effect is that the distribution reaching each layer stays in a range where activations are responsive and gradients are healthy, which allows higher learning rates and makes training much less sensitive to initialisation. It also adds mild regularisation, because each example's normalisation depends on the other examples in its batch. What it breaks is the independence of examples, and every cost follows from that: it behaves differently in training and inference, since inference uses running averages rather than batch statistics, so a bug in those averages produces a model that trains well and serves badly. It degrades at small batch sizes, and it does not fit sequence models cleanly, which is why layer normalisation is used there.

Randomly zeroing a fraction of units on each training step, so no unit can rely on the presence of any other and the network is forced to spread its representation rather than build fragile co-adapted paths. It is loosely an ensemble over exponentially many thinned networks that share weights, and at inference time all units are active with activations scaled so that expected magnitudes match. The costs are worth naming. Training is noisier and slower to converge, because each step optimises a different sub-network. The rate is another hyperparameter, and too much of it underfits — a model that never quite reaches the training performance it should. It interacts awkwardly with batch normalisation, since both inject batch-dependent noise. And on modern architectures with normalisation and heavy data augmentation, its marginal benefit is often small.

Because a recurrent network compresses everything it has read into one hidden state and processes tokens strictly in order, and attention removes both constraints. Any output position can look directly at any input position, so the path length between two related tokens is one step rather than the distance between them, which is what makes long-range dependencies learnable — the recurrent path multiplies gradients once per intervening token and the signal decays. And because each position's attention is computed independently, the whole sequence is processed in parallel during training, which turns a sequential dependency into a matrix multiplication and is the practical reason scale became possible. The costs are real: attention is quadratic in sequence length in both time and memory, and having discarded order it needs positional information injected explicitly.

When the data is tabular and modest in size, when you must explain individual decisions, or when nobody will own the operational surface. On a few hundred thousand rows of heterogeneous columns, gradient-boosted trees usually win on accuracy and always win on effort — there is no architecture to choose, training takes minutes on a laptop, and the result is stable across seeds. Neural networks earn their place where the input has structure that weight sharing exploits, such as images, audio, text and sequences, or where you want to fine-tune a pretrained model and inherit representations you could never learn from your own data. The costs to state plainly are hyperparameter sensitivity, non-determinism that complicates reproducibility, hardware, and an artefact that is far harder to debug when it starts behaving oddly in production.

That the statistical properties you are relying on — most importantly the mean and variance — do not change with time. It matters because almost every classical forecasting method assumes it, and fitting one to a trending series produces a model that has learned the trend as a level and will be confidently wrong the moment the trend continues. Differencing removes a trend, seasonal differencing removes a fixed seasonal pattern, and a log transform stabilises variance that grows with the level. The subtler point is that stationarity is a property of the model's residuals in practice: if what remains after you have modelled trend and seasonality still drifts, your model is missing a component. And a series that is not stationary in a way you cannot transform away is telling you the process itself changed, which is a data question rather than a modelling one.

Every quantity used to produce a forecast for a date must have been knowable before that date, and the ways to violate that are all invisible in the metric.

Forecasting daily demand 14 days ahead. Data: 2024-01 to 2026-06.

Wrong: random 80/20 split of the days
  the model trains on July and predicts June
  it also trains on the day either side of every test day, so it
  interpolates rather than forecasts
  reported MAPE 4.1%, production MAPE 19%

Wrong in a subtler way: correct time split, but
  features scaled using the mean of the whole series
  holiday flags built from a calendar file created in 2026
  a "trailing 28-day average" that includes the target day

Right: rolling origin evaluation

  origin      train on            forecast          horizon
  ----------  ------------------  ----------------  -------
  2025-09-30  start .. 2025-09-30  2025-10-01..14   1..14
  2025-10-14  start .. 2025-10-14  2025-10-15..28   1..14
  2025-10-28  start .. 2025-10-28  2025-10-29..11   1..14
  ...
  2026-06-16  start .. 2026-06-16  2026-06-17..30   1..14

  Refit at each origin using only data up to it. Report error by
  horizon step, not just in aggregate.

The rolling origin is the whole method: it reproduces the operational reality of standing at a date, knowing only the past, and forecasting forward. Averaging across many origins gives an estimate that is not a single lucky fortnight, and it exposes whether accuracy is degrading over time.

Reporting error by horizon step is the part people skip and it is where the information is. A model may be excellent at one day ahead and useless at fourteen, and a single aggregate number hides that completely — while the business decision almost always depends on a specific horizon, because that is the lead time of whatever it is planning.

The subtle lookahead cases are the ones that survive review. Scaling with the full-series mean leaks the future's level into every training row. A holiday calendar assembled retrospectively encodes knowledge of which days turned out to be disrupted. And a trailing window that includes the target day is the same bug as target leakage, dressed as a time-series feature.

The baseline to beat is a seasonal naive forecast — this Tuesday equals last Tuesday. On strongly seasonal series it is hard to beat, and a model that does not clear it is not a forecasting model. Quoting a MAPE without that comparison says nothing about whether anything was learned.

By separating the components you can name from the ones you must learn. Multiple seasonalities usually coexist — day of week, day of month, week of year — and each should be represented explicitly, either as calendar features or as Fourier terms that let a smooth model express periodicity without one column per level. Holidays need a table rather than a rule, because they move, differ by region, and have effects that spill into adjacent days, so a flag on the day alone misses the shopping before and the lull after. Once-off events such as a promotion or an outage should be marked, or the model attributes their spike to whatever seasonal term happened to coincide. The cost of all this is that the calendar becomes a dependency you must maintain forwards in time, and a missing future holiday silently degrades every forecast past it.

You cannot score millions of items per request within a latency budget, so the architecture is two stages with different objectives.

flowchart LR
    U[Request context<br/>user and session] --> C[Candidate generation<br/>millions to hundreds]
    C --> R[Ranking model<br/>hundreds to tens]
    R --> B[Business rules<br/>and diversity]
    B --> D[Displayed slate]
    D --> L[Logged impressions<br/>clicks and skips]
    L --> C
    L --> R

The two stages optimise different things and this is the point most often missed. Candidate generation optimises recall at low cost: several cheap retrievers — embedding nearest-neighbours, co-occurrence, recent popularity, editorial picks — each contribute a few hundred plausible items, and the union is the candidate set. Ranking optimises precision at the top using an expensive model with far richer features, including cross-features between the user and the item that would be impossible to compute for millions of rows.

The consequence is a hard ceiling: anything the first stage misses cannot be recovered by the second, however good the ranker. So the first metric to instrument is candidate recall against a labelled set of items the user did engage with, and teams that spend six months on the ranker while the generator has 60% recall are optimising the wrong stage.

The rules layer after ranking exists because the ranker optimises one objective and a product has several — diversity so the slate is not five near-identical items, freshness, supplier fairness, and hard exclusions such as out-of-stock or age-restricted. Keeping these outside the model makes them auditable and changeable without a retrain, which is usually worth the loss of optimality.

The feedback arrows are the danger. Both stages are trained on logs the current system produced, so items the current system never showed have no engagement data, and the model learns that they are uninteresting. Breaking that loop needs deliberate exploration — a fraction of traffic showing candidates the ranker did not choose — and treating the cost of that exploration as the price of not going blind.

Position bias is that an item's click rate depends heavily on where it was shown, so the top slot collects clicks partly because it is the top slot. Train naively on click logs and the model learns to predict position rather than relevance, then promotes whatever the previous model promoted. The feedback loop compounds it: items never shown accumulate no positive data, so their estimated relevance stays low, so they are never shown, and the system's beliefs harden regardless of truth. The mitigations are to model the bias explicitly — inverse propensity weighting using the probability an item was examined at that position — to randomise position for a small slice of traffic so you can measure the bias rather than assume it, and to reserve exploration traffic for unproven items. All three cost short-term engagement to buy a model that can still learn.

Because offline evaluation grades the model on data produced by the system it is replacing. Four causes recur. The logs only contain outcomes for what was shown, so the new model's preferred items have no labels and are scored on an extrapolation. The model changes what users see and therefore what they do, which no static dataset can capture. Offline metrics measure the model while online metrics measure the product, including latency, layout and the rules layer — a ranker that is better and 80 milliseconds slower can lose. And the offline metric is often a proxy chosen for convenience, such as click rate standing in for long-term retention, so improving it can actively harm the real objective. Treating offline results as a screening gate and an experiment as the decision is the posture that follows.

A strong answer treats this as debugging with a short list of causes and says how to separate them. Was the offline number ever real — a leaking feature, a split that ignored time or groups, a test set looked at repeatedly? Is the deployed model scoring what the offline model scored, which you establish by logging the serving feature vector and comparing its distribution against training? Did the threshold survive deployment, since a good ranker at the wrong cut-off produces alerts nobody can process? Does the prediction reach a decision at all, because a score in a queue the team ignores changes nothing by construction? And was the offline metric only ever a proxy, so the model can gain AUC and lose revenue? The answer states the order it would check these in, cheapest first, and the evidence that settles each.

A weak answer reaches for the model — a different algorithm, more features, more tuning — none of which addresses any of those causes, and all of which consume a quarter. The tell is that no question is asked about the deployment, the label, the operating point or the decision.