Your offline ranking metrics improved but engagement dropped in the A/B test. What happened?

Check the experiment before rewriting the model

Before reaching for theory, rule out the boring causes, because they are common and they are cheap to eliminate. Confirm the two arms were assigned properly and are comparable on pre-period metrics. Confirm the drop is outside the range you would see between two identical arms — if you have never run an A/A test on this surface, you do not know what that range is. Check latency, because a heavier ranker that adds sixty milliseconds to a page can cost more engagement than any relevance gain recovers, and that shows up as a relevance failure when it is an infrastructure one. Check for a broken feature in the serving path, where a feature computed one way in training arrives differently at request time.

Having cleared those, the explanation is nearly always that the offline evaluation was measuring something other than what you thought.

Position is confounded with relevance

Your logs record that an item at position one was clicked and an item at position nine was not. The tempting reading is that the first item was more relevant. But users scan from the top, attention decays sharply with depth, and many users never reach position nine at all. So the click record reflects the sum of two things — how relevant the item was and how much attention its slot received — and the log does not separate them.

Now train a ranker on those clicks as labels. The strongest pattern in the data is that items the previous ranker put near the top were clicked, so the fastest way to fit the data is to learn to reproduce the previous ranker's ordering. Your model does that, offline NDCG rises because NDCG is computed against those same clicks, and you have built an accurate imitation of the system you were trying to improve. Live, it does nothing, or it does slightly worse because the imitation is imperfect in ways the old system's genuine signal is not.

The consequence is sharper than it first appears: an offline metric computed on logged clicks is partly a similarity measure between your candidate ranker and the incumbent. A model that reorders aggressively will look worse offline whether it is better or not.

What the logs do not contain

Position bias is about the items you showed. The deeper problem is the items you did not.

The impression log contains only what the previous system surfaced, and the candidate set was chosen by the previous retrieval stage. Items outside it have no impressions and therefore no labels, so every offline metric is silent on them. Any improvement your new model makes by surfacing something the old one never showed is invisible offline — it cannot be credited, because there is no record of what the user would have done.

Repeat that over several model generations and the effect compounds. Each model is trained on data generated by the last, the surfaced slice of the catalogue narrows, and offline metrics rise steadily throughout because they are measured against a shrinking, self-selected sample. This is why a model can look like a clear win on every offline number and reduce engagement: it has optimised agreement with the past on the region of the catalogue the past already covered.

Two related traps sit alongside it. A missing click is not a negative label — the user may not have seen the item, and treating unexamined items as negatives teaches the model that everything below the fold is bad. And a click is not the outcome anyone wants: if you optimise clicks and the live metric is time spent or retention, a model that promotes clickable but disappointing items will improve one and damage the other, exactly as observed.

Corrections, and what each one assumes

The direct fix for position is to model it explicitly. Estimate a per-position examination probability, then weight each logged observation by the inverse of that probability so that a click at position nine, which required an unlikely amount of attention, counts for more than a click at position one. That is inverse propensity weighting, and it will produce an unbiased estimate provided your propensities are right, no propensity is so small that dividing by it explodes the variance, and no unlogged factor influenced both what was shown and what was clicked. Say those conditions when you propose it, because the assumptions are where it fails.

Getting the propensities requires an intervention rather than more analysis. The usual route is deliberate randomisation: on a small share of traffic, swap adjacent positions or shuffle the top few slots. Because the item in a slot is then independent of the slot, the difference in click rate across positions estimates the attention curve directly. That randomised slice is expensive and it is the only data in your warehouse that is genuinely unconfounded, which makes it the most valuable logging you can commission.

You can also fold position into the model. Include position as a feature during training so the model attributes part of each click to the slot, then serve with that feature held at a constant value so the score reflects relevance alone. It is cheaper than randomisation and it assumes the position effect is separable from everything else, which is an approximation.

Independently of the correction, log the propensity at serving time. A ranker that records the score and the selection probability for every impression makes every future offline evaluation better, and it cannot be reconstructed after the fact.

How to read offline results after this

Treat offline evaluation as a filter, not a verdict. It is good at catching a model that is broken and unreliable at telling you which of two working models is better, because the ordering it prefers is contaminated by the policy that produced the data.

So change what you ask of it. Report offline metrics on the randomised slice as well as the full log, and trust the former when they disagree. Look at how far the new ranking diverges from the old, and read a large divergence as a reason to test rather than a defect. Evaluate on the most recent window rather than a long history, since the confound is strongest where your own policy has been stable. And decide before the test which live metric is the decision variable and what trade you will accept between clicks and time spent — that conversation held afterwards becomes an argument about which number to believe, and the answer stops being empirical.