Machine learning fundamentals

When the rule is already known, when there is no label, or when nobody will act on the output. A deterministic policy — block the payment if the country is on a sanctions list — should be an if statement, because a model would learn the rule imperfectly and make it unauditable. A problem with no recorded outcome is not a supervised problem at all until somebody starts recording it, and pretending otherwise produces a model fitted to a proxy nobody agreed to. And a prediction that changes no decision is a report, not a model, however accurate it is. The diagnostic question is what happens differently on the day the model is right; if the answer is "we look at a dashboard", the model was never the work.

A label you can obtain at scale and trust, the same features at prediction time that you have in training, and a decision the prediction changes. Each one kills projects independently. The label is the most common failure: it arrives months late, or it is produced by inconsistent human judgement, so the model learns the annotators rather than the phenomenon. Prediction-time availability is the quietest failure, because the warehouse contains fields that were written after the event you are predicting and nothing in the schema says so. And the decision determines everything downstream — the metric, the threshold, the latency budget, the acceptable error direction. Establishing all three takes a conversation, not a notebook, and doing it last is how six months disappear.

The model is only the middle of the loop, and every arrow in it is a place the two halves can disagree.

The single feature definition feeding both paths is the load-bearing part of this diagram. If training computes a thirty-day average in warehouse SQL and serving computes it in application code, you have two features with one name, and the model is being asked questions in a dialect it was not taught. Nothing alerts, because offline evaluation uses the offline definition.

as of prediction time on the first node is the other constraint that gets dropped. A training row dated June may only use data that existed in June, which means the join is point-in-time rather than a straightforward equality on customer identifier. Most leakage is a violation of that one arrow.

The threshold travels with the model, not with the code that calls it. A classifier emits a score; the number that turns a score into an action is a business decision, and versioning it separately from the model is how a retrain silently changes the alert volume the operations team receives.

The loop closing at the bottom is what makes the system improvable and what makes it dangerous. Outcomes recorded on scored requests become the next training set, so the model's own behaviour shapes the data it will next learn from — you only observe outcomes for the cases you acted on, and the cases you rejected have no label at all.

To establish what the number you report actually means, because a metric with no reference point is unreadable. Three baselines are worth having. The trivial one — predict the majority class, or predict last week's value — tells you whether the model has learned anything at all, and on imbalanced or strongly autocorrelated data it is embarrassingly strong. The existing process, whether that is a rules engine or a human team, is the one the business compares against, and beating a model that does not exist is not a result. And a simple model such as regularised logistic regression tells you how much of the achievable signal is linear, which sets your expectation for how much a complicated model can add. Reporting the baseline unprompted is a reliable competence signal.

By working backwards from the action, then checking which candidate target you can actually observe. If the action is offering a retention discount, "will churn within ninety days" is a better target than "is dissatisfied", because the first is a recorded event with a date and the second is an unmeasurable state. Then check the horizon against how long the intervention needs: predicting churn one day out is accurate and useless, because there is no time to act. Then check the base rate, since a target so rare that a month of data holds forty positives will not support a model regardless of technique. The target that survives all three is often narrower and more boring than the one the business asked for, and saying so is the job.

It sets the clock on everything. If a loan default is only observable after ninety days, then your freshest usable training data is ninety days old, your first honest evaluation of a newly shipped model is ninety days away, and any concept drift is discovered a quarter after it started. That forces a two-tier monitoring design: feature and prediction distributions watched immediately as an early warning, true performance measured against ground truth when it arrives. It also argues for a short-horizon proxy target you can measure weekly, used strictly for drift detection rather than for the decision. The cost people miss is experimental: with a ninety-day label, you get four honest model comparisons a year, so the sequence in which you try things matters far more than usual.

Because it decides what a row is, and therefore what the metric measures and what the model can possibly learn. Predicting "will this machine fail" per machine per month, per machine per day, and per component per day are three different problems with different base rates, different feature windows and different usable actions. Get it wrong and every number afterwards is coherent and meaningless: a per-day model evaluated per month will look far better than it is, because one correct day out of thirty counts as a caught failure. The unit also fixes the independence assumption behind your evaluation — many rows per machine means rows are correlated, and a random split will put the same machine on both sides. Declare the unit before choosing anything else.