The modelling discipline: turning a business decision into a prediction problem, building features that will exist at prediction time, and choosing a metric that corresponds to the decision. Most failures here are framing and leakage failures, not algorithm failures.
High demand22 min readMachine Learning Engineer, Data Scientist, Applied Scientist, Quantitative Analyst, Research Engineer, Analytics Engineer moving into modellingUpdated 2026-07-28
Assumes you know: Comfortable writing Python and manipulating tabular data, Basic probability and statistics - distributions, expectation, conditional probability, Linear algebra to the level of matrices and dot products
Overview
What this area actually covers
Machine learning, as a discipline you would be interviewed on, is the work of turning a
decision somebody has to make into a prediction problem, learning a function that makes that
prediction from historical data, and establishing honestly how well it does — where honestly
means "on data it has not seen, measured by something that corresponds to the decision, at
the operating point you will actually run".
That definition puts most of the weight before any algorithm is chosen, which is where the
weight belongs. The choice between logistic regression and gradient-boosted trees is a real
decision with real consequences, and it is not usually what determines whether a project
succeeds. What determines that is whether the target variable corresponds to the decision,
whether the features will be available at the moment the prediction is needed, whether the
evaluation split respects the structure of the data, and whether the metric being optimised
is the one the business cares about. A project can fail every one of those tests and still
produce a model with an impressive score, which is precisely why the score is not the
deliverable.
The scope covered here runs from the classical supervised methods through feature
construction, evaluation, the specific problem of rare events, unsupervised and anomaly
detection where there are no labels at all, the foundations of neural networks, forecasting
over time, and ranking and recommendation. What is deliberately excluded is worth naming.
Moving and shaping data at scale — pipelines, warehousing, Spark, orchestration — is data
engineering and lives in Data & AI Engineering. Everything required to
get a trained model into production and keep it working is
MLOps & ML Platform. And building product features on top of
a language model you did not train is a separate discipline again, in
Generative AI & Large Language Models.
That last boundary needs stating plainly because the vocabulary has blurred. Prompting a
large model is not machine learning in the sense examined here: there is no training set, no
held-out split, no metric selection tied to a loss function, and no leakage risk of the kind
that dominates this section. Both are real disciplines. They are interviewed separately,
hired separately, and confusing them wastes preparation. The overlap is genuine only in two
places — evaluation discipline, and the deep learning foundations that explain what a
transformer is doing.
A note on this site's structure. The older Data & AI Engineering
section still carries ML Fundamentals and Statistics & Experimentation subsections from
earlier waves, and those pages remain. ML Fundamentals is a compressed treatment of what
this section now covers at length. Statistics & Experimentation — hypothesis testing, A/B
design, reading a result honestly — has deliberately not been duplicated here, so if you
need that material it is still the right place to go.
The order runs from the methods, through the two subsections that cause most real failures,
into the specialisations. Read feature engineering and leakage, and model evaluation, before
anything else — not because they are introductory but because they are where the mistakes
happen, and because the remaining subsections are hard to judge without them. The last three
are separate because their evaluation rules differ from the general case, which is the whole
reason they need separating.
Subsection
What it is for
Supervised Learning
Choosing and fitting a model family for a labelled problem
Feature Engineering & Leakage
Building inputs that exist at prediction time and do not encode the answer
Model Evaluation & Metrics
Measuring in a way that survives contact with the decision
Imbalanced & Rare Events
Modelling the events that matter and are almost never observed
Unsupervised & Anomaly Detection
Finding structure and outliers with no labels to learn from
Deep Learning Foundations
The mechanics behind neural networks, and when not to use one
Time Series & Forecasting
Prediction where order matters and a random split is invalid
Recommenders & Ranking
Two-stage retrieval and ranking, and the feedback loops it creates
Supervised Learning
Linear and logistic models, decision trees, random forests and gradient boosting,
regularisation and what it is actually doing to the coefficients, and the bias-variance
reasoning that connects model capacity to generalisation. Its purpose in the section is to
give you a defensible answer to "why this family", and the defensible answer comes from the
data and the constraints rather than from habit: the number of rows, the number and type of
features, whether you need a monotonic or explainable relationship, whether the deployment
target can carry the model, and how much tuning time exists. Interviewers ask why you would
start with logistic regression on a tabular problem, and the strong answer is not that it is
simple but that it is a calibrated, interpretable, fast baseline whose failure tells you
something.
Feature Engineering & Leakage
Encoding categorical variables, scaling, interaction and ratio features, and the treatment
of missing values as information rather than as a nuisance — and then the part that matters
more than all of it, which is leakage. Target leakage, where a feature encodes the label.
Temporal leakage, where a feature carries information from after the prediction moment.
Train-serve skew, where the feature computed offline is not the feature computed in
production. This is its own subsection and it is the most important one on the page, because
leakage is the failure that looks like success: the model scores beautifully in validation,
ships, and performs at chance. Everyone who has worked in this field has done it at least
once, and the ability to describe the time you did is a genuinely strong interview answer.
Model Evaluation & Metrics
Cross-validation schemes that respect the structure of the data — grouped, stratified,
time-ordered — rather than shuffling indiscriminately; the difference between ROC and
precision-recall curves and when each misleads; calibration, meaning whether a predicted
probability of 0.7 corresponds to 70 per cent of such cases occurring; threshold selection
as a business decision rather than a default of 0.5; and tying a metric back to what the
decision costs. Separate from the supervised subsection because the measurement is a skill
independent of the model, and because most interview questions that appear to be about
algorithms are actually about this.
Imbalanced & Rare Events
Fraud, equipment failure, churn at low base rates, adverse medical events — problems where
the positive class is a fraction of a per cent. It covers why accuracy is meaningless there,
what resampling and class weighting actually change, why the choice matters for calibration,
and how to evaluate at the operating point you will run rather than across the whole curve.
It exists as its own area because the general evaluation intuitions break down: a model that
predicts "no" for everything can be 99.9 per cent accurate, precision and recall trade in
ways that map directly onto operational capacity, and the real question is usually "how many
alerts can the team review in a day" rather than which model scored highest.
Unsupervised & Anomaly Detection
Clustering and the genuinely hard problem of evaluating it, dimensionality reduction for
both modelling and inspection, density and isolation-based outlier methods, and detecting
anomalies where you have no labels at all. It is separate because the absence of a label
changes everything downstream: there is no held-out score to appeal to, so validation becomes
a matter of stability, of domain review, and of whether the output supports a decision. Many
real detection systems — intrusion, fraud, machine health — begin here because labels do not
exist yet, and the reasoning about how to bootstrap towards labels is a large part of the
material.
Deep Learning Foundations
Backpropagation as a mechanism you can explain rather than invoke, optimisers and how
learning rate behaviour manifests as a training curve, normalisation and regularisation,
attention and the transformer, and — given equal weight — when a neural network is the wrong
choice. This subsection sits deliberately after the classical material because the honest
position on tabular problems is that gradient-boosted trees remain a very strong default and
a neural network is usually more work for less. Where deep learning is not optional is
unstructured data: text, images, audio, and anything where the representation must be learned
rather than engineered. It also earns its place because it is the foundation under the
generative AI sections, and a reader who understands attention here will read those with far
less mystification.
Time Series & Forecasting
Stationarity and seasonality, decomposition, backtesting with a rolling origin, the effect of
forecast horizon on both method and achievable accuracy, hierarchical forecasts that must
reconcile across levels, and the reason a random train-test split silently invalidates a
forecasting model. Its own subsection because the evaluation rules are different in a way
that catches out strong general practitioners: shuffling a time series lets the model learn
from the future, and the resulting score is not merely optimistic but meaningless. Demand
planning, capacity forecasting and financial projection all live here, and they are among the
most common real applications of modelling in ordinary businesses.
Recommenders & Ranking
Collaborative and content-based signals, the two-stage architecture of candidate generation
followed by ranking, the cold-start problem for new users and new items, position bias and the
feedback loop created by training on logs your own model produced, and the frequent divergence
between an offline metric improvement and an online result. Separate because it is the most
commercially deployed form of machine learning and because it has a structural problem the
rest of the section does not: your training data is generated by your previous model, so the
system is learning from its own behaviour. Understanding that loop, and what randomisation
or logged propensities do about it, is the distinguishing knowledge here.
Where it sits in a real system
The chain from a business decision to a deployed model is short, and every link in it is a
place where projects go wrong. Tracing it in the correct direction — decision first, model
last — is the habit that separates practitioners from people who start with a dataset.
Loading diagram…
The link that carries the most weight is the third, and it is the one usually skipped. Fixing
the prediction moment — the instant at which the model must produce an answer — determines
which features are legitimate, and every leakage bug in existence is a feature that violates
it. The loop closing back to the decision is the second thing to notice: a threshold is not a
modelling parameter but a statement about how many false positives the business will accept,
which means it cannot be chosen without the people who handle them.
The practical consequence is that the deliverable of modelling work is rarely a model file.
It is a model, plus a stated operating point, plus the evidence that the point is achievable
on data resembling production, plus a statement of what happens to the decision when the model
is wrong in each direction. Candidates who present the first item and none of the rest read as
junior regardless of the score they achieved.
Tying a metric to a decision
Metric selection is asked in almost every modelling interview and is usually answered as a
vocabulary question. It is a business question. The route to a defensible answer is to name
what each kind of error costs, then choose the measurement that reflects that asymmetry.
Situation
The asymmetry
Metric that reflects it
The wrong default
Fraud screening with a review team
A missed fraud costs money, a false alert costs review time
Precision at the k the team can review
Accuracy, which the base rate flatters
Predictive maintenance
An unplanned failure costs far more than an early inspection
Recall at an acceptable alert rate
F1, which weights both errors equally by fiat
Credit decisioning
Probabilities feed a pricing calculation
Calibration plus a ranking measure
Any threshold metric alone
Demand forecasting
Over and under forecasting have different costs
An error measure matching the cost, by horizon
A single average error across all horizons
Search and recommendation
Only the top few results are seen
A rank-weighted measure over the visible list
Accuracy over the whole candidate set
Rare medical screening
A missed case is severe, a false alarm triggers a cheap follow-up
High recall with the false-positive rate stated
ROC area, which looks strong at any base rate
The row that generalises furthest is the third. Where the model's output feeds a downstream
calculation rather than a binary action, calibration matters more than any threshold metric,
because a well-ranking but badly calibrated model produces the right ordering and the wrong
numbers — and the downstream calculation cannot tell.
Leakage, worked
Leakage deserves a worked example because it is the failure most likely to reach production
and the one most often described abstractly. Consider a churn model for a subscription
business, predicting whether a customer will cancel in the next thirty days.
Three lessons sit in that table. The obvious leaks are easy and are not the dangerous ones;
cancellation_reason_code will be spotted by anybody. The dangerous ones are the CAREFUL
rows, where a perfectly reasonable feature becomes a leak depending on when it was
computed, which is invisible in the column name and visible only in how the training set
was assembled. And the split matters as much as the features: a random row split on customer
data lets the model memorise customers and lets the future inform the past, so it produces a
number that means nothing. This is exactly the problem that point-in-time correctness in
MLOps & ML Platform exists to solve systematically.
Who does this work
The two dominant titles are machine learning engineer and data scientist, and the difference
between them is largely a difference in where the work stops. A data scientist's output is
typically an analysis, a model and a recommendation, delivered to somebody who will act on
it; a machine learning engineer's output is a model running in a system, which means the
job includes the pipeline, the serving path and the monitoring. Both are modelling roles.
Only one is usually accountable at three in the morning, and that difference shows up in how
each interviews: the data scientist loop leans on statistics, experiment design and
communication, and the engineer loop adds systems, pipelines and production reasoning.
Applied scientists sit closer to method development, usually in organisations large enough to
have problems the standard toolbox does not solve, and their loops include research judgement
and often a publication record. Quantitative analysts in finance do statistically demanding
modelling under a regulatory regime that constrains which models are permissible and requires
that decisions be explainable, which is a genuinely different set of constraints. Research
engineers build the tooling and infrastructure that lets scientists run experiments, which is
adjacent to platform work. And analytics engineers moving into modelling arrive with an
underrated advantage — they already know where the data comes from and where it lies, which is
where most modelling errors originate.
A week in modelling work is less algorithmic than the field's reputation. You spend most of it
on data: establishing what a field actually means, discovering that a value changed definition
in a previous year, finding that the label you were given was assembled with hindsight. Then
you build a baseline, which frequently performs suspiciously well and turns out to be leaking.
Then you argue about a threshold with the team who will handle the alerts. Then you write the
document explaining what the model does and does not do, which is the artefact that survives.
Demand, adoption and how that is changing
Demand is high and has an unusual shape at present, so it is worth being precise. Attention
and new funding have moved sharply towards generative AI, which has made classical modelling
look quieter than it is. What is actually happening is that the classical work has become
infrastructural: fraud detection, credit risk, pricing, demand forecasting, churn, ranking and
recommendation are load-bearing systems in ordinary companies, they are not going to be
replaced by a language model, and they require people who can maintain and improve them.
Hiring for that is steady, less fashionable, and considerably less crowded at the experienced
end than the generative roles.
Three specific forces are visible. Tabular problems have converged on gradient boosting plus
careful feature work, which means the differentiating skill has moved decisively from
algorithm knowledge to framing, data quality and evaluation — and interviews have followed,
which is why so many modelling loops now consist of a leakage question, a metric question and
a case study rather than a derivation. Automated tooling has commoditised the fitting and
tuning step without touching the framing step, so the value has concentrated where the tooling
cannot reach. And regulation is expanding in the domains that use modelling most heavily,
which raises the premium on explainability, documentation and fairness testing, covered in the
governance material in MLOps & ML Platform.
The honest counterweight is about the entry level. Junior modelling roles are genuinely
competitive: the supply of people who have completed courses and Kaggle competitions is large,
and the skill those select for — squeezing a metric on a clean, correctly split, already-framed
dataset — is the part of the job that has been most automated. What is scarce is someone who
can take a vague business problem, notice that the proposed label is unmeasurable, define one
that is, and say what the model should do when it is uncertain. That is a judgement skill, and
it is why practitioners with domain experience often move into this work more easily than
people who arrive from a purely technical route.
What makes it hard
Framing is the hardest part and nobody teaches it. Every course hands you a dataset, a
target column and a metric. Every real project begins with a business problem that has none of
those, and the work of deciding what to predict, at what moment, with what label definition, is
both the highest-leverage and least-practised skill in the discipline. Get it wrong and
everything downstream is well-executed and useless: a model predicting whether a customer
has churned is a different and far less useful thing than one predicting whether they will,
and the difference is one sentence in a specification.
Leakage is the failure that looks like success. No other common bug in engineering produces
a result that everyone celebrates. A leaking model validates at a level that should itself be
the warning sign, and the discipline of treating an unexpectedly strong score as suspicious
rather than gratifying takes deliberate cultivation. The subtle forms — a feature computed as
of today rather than as of the prediction date, an aggregate whose window overlaps the label
period, a validation split that puts the same entity on both sides — are invisible in the
feature list and visible only in how the dataset was assembled.
Metric choice is a business decision made by technical people. Optimising a metric that
does not correspond to the decision produces a model that is provably better and practically
worse. The specific traps recur: accuracy on an imbalanced problem, ROC area when the base rate
is tiny, F1 when the two errors have wildly different costs, an average error across forecast
horizons that matter differently. Behind all of them is the same missing step — nobody asked
what each kind of mistake costs.
Rare events break the intuitions that work everywhere else. When the positive class is a
fraction of a per cent, most of the standard reasoning becomes unsafe. Resampling changes the
model's calibration, so the probabilities it emits no longer mean what they say unless you
correct for it. Cross-validation folds may contain almost no positives, making the variance in
your estimate larger than the difference you are trying to measure. And the operational
constraint usually dominates the statistical one: the useful question is how many alerts a team
can act on, which fixes the operating point before any modelling begins.
Offline and online results diverge, and the reasons are structural. A model that improves an
offline metric can fail to improve anything in production, and the causes are enumerable: the
production feature values differ from the training ones, the population has shifted since the
training window, the model's own output changes user behaviour and therefore the data it next
learns from, and the offline metric measured something correlated with but not equal to the
outcome. Recommender systems suffer this most acutely because they train on logs their
predecessor produced. Expecting the divergence, and designing an online test that can detect it,
is what experience buys.
Most of the work is data work, and it is not what people signed up for. The proportion of
modelling time spent establishing what a field means, reconciling definitions that changed,
handling missing values that are missing for informative reasons, and reconstructing what was
knowable at a point in the past is high and consistent across employers. Someone whose interest
in the discipline is algorithmic will find this a disappointment; someone who is curious about
the domain will find it the interesting part.
Why study it
Study it if you like problems where the difficulty is in the reasoning rather than the tooling,
and if you are comfortable with an answer that is a probability rather than a fact. Study it if
you want to work on decisions rather than features: modelling is unusual in that its output is
directly a business action — this transaction is declined, this machine is inspected, these
results are ordered this way — and the closeness to consequence is what makes it satisfying.
There is also a strong argument for studying this material even if you intend to work in
generative AI. Evaluation discipline transfers directly and is the scarce skill there.
Understanding held-out data, calibration, base rates and metric choice is what lets you tell
whether an AI feature actually improved. And the deep learning foundations here are what make
transformer behaviour comprehensible rather than magical. A generative AI engineer who has done
classical modelling is measurably better at the part of that job that is hardest.
Be honest about who should not prioritise it. If you want to build product features on top of
existing models, this is a longer route than
Generative AI & Large Language Models and much of the classical material
will not appear in your interviews. If your interest is in the infrastructure — pipelines,
serving, scaling — you will be happier and more employable in
MLOps & ML Platform, which is less crowded. And if you are
looking for the fastest route into a technical career, this is not it: the entry level is
competitive, the interviews probe judgement that comes from doing the work, and a portfolio of
competition notebooks does not demonstrate the skill that is scarce.
Your first hour
Take a dataset you understand from your own work or life — not a benchmark dataset, because the
whole exercise depends on your knowing what the columns mean — and do the framing work before
touching a model.
Write down five things, in this order. First, the decision: what would somebody do differently
if they had this prediction. Second, the prediction moment: at what instant must the answer
exist. Third, the label: how exactly is the outcome defined and observed, and how long after the
prediction moment does it become known. Fourth, the features, and for each one a single word
saying whether it is knowable at the prediction moment. Fifth, the cost of each kind of error,
in whatever unit is natural — money, time, risk.
Only then fit something, and fit the simplest thing: a logistic regression or a small decision
tree, with a split that respects the structure you identified in step two. Record two numbers,
the metric you chose in step five and a naive baseline such as always predicting the majority
class or persisting the last value.
The last two lines are the artefact. A score that falls when you remove leakage is the most
instructive result available to a beginner in this field, and being able to narrate that
sequence — the too-good number, the feature that caused it, the honest number — is a better
interview answer than any leaderboard position.
If you have longer, change only the threshold and watch precision and recall move against each
other. Then ask which point the retention team would choose, and notice that the question is not
answerable without them.
What this is not
It is not artificial intelligence in the sense the phrase now carries. This section is about
learning functions from data with labels, structure and metrics, and the fact that generative
models are also machine learning does not make the two the same job. Interviews, teams and
career paths are separate.
It is not algorithm implementation. Nobody in a normal role writes gradient boosting from
scratch, and while implementing one is an excellent way to understand it, being able to derive
the update rule is not the skill being hired. Interviewers who ask you to explain how a method
works are checking whether you know what it assumes and where it fails, not testing recall of a
derivation.
It is not data engineering. Building the pipelines that deliver data is a distinct and
substantial discipline with its own hiring loop, covered in
Data & AI Engineering. The frequent overlap in practice — modellers doing
their own extraction — does not merge the skill sets, and a modelling interview will not credit
your Spark tuning.
It is not deployment. Getting a model into production, versioned, monitored and retrained is
MLOps & ML Platform. Plenty of good modellers cannot do it, and
plenty of good platform engineers cannot model. Knowing which of the two a role actually wants,
before the interview, saves a great deal of misdirected preparation.
And it is not a competition to maximise a metric. Competitions reward squeezing a score from a
fixed, clean, correctly split dataset, which is the one part of this work that is both already
solved and already automated. The scarce ability is deciding what to predict, noticing when a
score is too good, and saying what the model should do when it does not know.
The most useful reflex in this discipline is suspicion of a good result. Every leaking model
validated beautifully, and every practitioner has shipped one.
Candidate feature Verdict Why
tenure_days fine Known at prediction time
support_tickets_last_90d fine Historical window, ends before the moment
plan_tier fine Current state
cancellation_reason_code LEAKAGE Onlyexists once they have cancelled
last_login_days_ago CAREFUL Fine if computed asof the prediction date
Leaks if computed asof today forallrows
downgraded_in_last_30d CAREFUL Legitimate signal, but check the timestampis the event datenot the record updatedate
account_closed_flag LEAKAGE A near-copyof the label
avg_monthly_spend CAREFUL Leaks if the average includes months
after the prediction date
refund_issued LEAKAGE Usually happens as part of cancelling
Validation split: random 80/20byrow WRONG - the same customer appears inboth
sides, and later months predict earlier ones
Validation split: bydate, train before RIGHT-matches how the model will be used
a cutoff and test after
Decision Which accounts the retention team calls this week
Moment Monday 06:00, using data through Sunday 23:59
Label No active subscription 30 days later - known 30 days late
Cost A wasted call costs 10 minutes. A missed churn costs onemonth revenue
Operating pt Team can call50 accounts per week, so precisionat50is the metric
Baseline call the 50 longest-tenured at-risk accounts precision0.18
Model logistic regression, date-based split precision0.31
Leakage check- dropped 2 features that were computed asof today, precision fell
from0.62to0.31, which is the real number