Check the experiment before rewriting the model
Before reaching for theory, rule out the boring causes, because they are common
and they are cheap to eliminate. Confirm the two arms were assigned properly and
are comparable on pre-period metrics. Confirm the drop is outside the range you
would see between two identical arms — if you have never run an A/A test on this
surface, you do not know what that range is. Check latency, because a heavier
ranker that adds sixty milliseconds to a page can cost more engagement than any
relevance gain recovers, and that shows up as a relevance failure when it is an
infrastructure one. Check for a broken feature in the serving path, where a
feature computed one way in training arrives differently at request time.
Having cleared those, the explanation is nearly always that the offline evaluation
was measuring something other than what you thought.
Position is confounded with relevance
Your logs record that an item at position one was clicked and an item at position
nine was not. The tempting reading is that the first item was more relevant. But
users scan from the top, attention decays sharply with depth, and many users never
reach position nine at all. So the click record reflects the sum of two things —
how relevant the item was and how much attention its slot received — and the log
does not separate them.
Now train a ranker on those clicks as labels. The strongest pattern in the data is
that items the previous ranker put near the top were clicked, so the fastest way to
fit the data is to learn to reproduce the previous ranker's ordering. Your model
does that, offline NDCG rises because NDCG is computed against those same clicks,
and you have built an accurate imitation of the system you were trying to improve.
Live, it does nothing, or it does slightly worse because the imitation is
imperfect in ways the old system's genuine signal is not.
The consequence is sharper than it first appears: an offline metric computed on
logged clicks is partly a similarity measure between your candidate ranker and the
incumbent. A model that reorders aggressively will look worse offline whether it is
better or not.
What the logs do not contain
Position bias is about the items you showed. The deeper problem is the items you
did not.
The impression log contains only what the previous system surfaced, and the
candidate set was chosen by the previous retrieval stage. Items outside it have no
impressions and therefore no labels, so every offline metric is silent on them.
Any improvement your new model makes by surfacing something the old one never
showed is invisible offline — it cannot be credited, because there is no record of
what the user would have done.
Repeat that over several model generations and the effect compounds. Each model is
trained on data generated by the last, the surfaced slice of the catalogue narrows,
and offline metrics rise steadily throughout because they are measured against a
shrinking, self-selected sample. This is why a model can look like a clear win on
every offline number and reduce engagement: it has optimised agreement with the
past on the region of the catalogue the past already covered.
Two related traps sit alongside it. A missing click is not a negative label — the
user may not have seen the item, and treating unexamined items as negatives teaches
the model that everything below the fold is bad. And a click is not the outcome
anyone wants: if you optimise clicks and the live metric is time spent or
retention, a model that promotes clickable but disappointing items will improve one
and damage the other, exactly as observed.
Corrections, and what each one assumes
The direct fix for position is to model it explicitly. Estimate a per-position
examination probability, then weight each logged observation by the inverse of that
probability so that a click at position nine, which required an unlikely amount of
attention, counts for more than a click at position one. That is inverse propensity
weighting, and it will produce an unbiased estimate provided your propensities are
right, no propensity is so small that dividing by it explodes the variance, and no
unlogged factor influenced both what was shown and what was clicked. Say those
conditions when you propose it, because the assumptions are where it fails.
Getting the propensities requires an intervention rather than more analysis. The
usual route is deliberate randomisation: on a small share of traffic, swap adjacent
positions or shuffle the top few slots. Because the item in a slot is then
independent of the slot, the difference in click rate across positions estimates
the attention curve directly. That randomised slice is expensive and it is the only
data in your warehouse that is genuinely unconfounded, which makes it the most
valuable logging you can commission.
You can also fold position into the model. Include position as a feature during
training so the model attributes part of each click to the slot, then serve with
that feature held at a constant value so the score reflects relevance alone. It is
cheaper than randomisation and it assumes the position effect is separable from
everything else, which is an approximation.
Independently of the correction, log the propensity at serving time. A ranker that
records the score and the selection probability for every impression makes every
future offline evaluation better, and it cannot be reconstructed after the fact.
How to read offline results after this
Treat offline evaluation as a filter, not a verdict. It is good at catching a model
that is broken and unreliable at telling you which of two working models is better,
because the ordering it prefers is contaminated by the policy that produced the
data.
So change what you ask of it. Report offline metrics on the randomised slice as
well as the full log, and trust the former when they disagree. Look at how far the
new ranking diverges from the old, and read a large divergence as a reason to test
rather than a defect. Evaluate on the most recent window rather than a long
history, since the confound is strongest where your own policy has been stable.
And decide before the test which live metric is the decision variable and what
trade you will accept between clicks and time spent — that conversation held
afterwards becomes an argument about which number to believe, and the answer stops
being empirical.
© 2026 Preptima. Originally published at preptima.com (opens in a new tab).