You have changed the ranking. How do you decide whether search got better, rather than just different?

You cannot measure better without writing down what better means

The reason this question defeats people is that they reach for a metric before defining the target. Relevance is not a property of a document; it is a relation between a document and an intent, and an intent that only the user has. So the first artefact is a set of judgements: for a sample of real queries, a set of candidate results, and for each pair a rating of how well that result serves the likely intent.

Make the ratings graded rather than binary, because binary loses the distinction the ranking exists to express. For a product catalogue, a scale from "exactly what was asked for" through "a reasonable substitute" and "the right category, wrong item" down to "irrelevant" lets a metric reward putting the perfect match first rather than merely putting something acceptable on the page. Sample the queries by traffic weight so the set reflects what people actually search for, then deliberately over-sample the tail, because head queries are the ones already tuned by hand and the tail is where a ranking change does its real damage.

Two practical constraints follow immediately. Judging every document against every query is impossible, so you judge a pool: the union of the top results from every system you are comparing, which guarantees that anything either candidate ranks highly has been rated. And judgements decay, because the catalogue turns over — a judgement set built a year ago rates items that no longer exist and misses the items that now win, so it will report that nothing has changed.

The offline metric has to be rank-aware

Once you have judgements, the comparison must reflect that position matters. Precision at k treats the tenth result as worth exactly as much as the first, which is not how anyone reads a page.

Discounted cumulative gain is the standard because it encodes both halves of the intuition: each result contributes its relevance grade, discounted by a function of its position, so moving a good result up is an improvement and burying it is a loss. Normalising against the discounted gain of the ideal ordering gives nDCG, a score between zero and one that is comparable across queries with different numbers of relevant documents — which matters, because without normalisation a query with twenty good answers dominates one with two. Reciprocal rank is the right metric instead when there is exactly one correct answer, as with a search that is really a lookup, since then all you care about is how far down the single answer sat.

The honest limitation of all of this is that offline evaluation measures agreement with your judgements, not user satisfaction. If the judges misunderstand the intent behind an ambiguous query, the metric will faithfully reward a ranking that satisfies the misunderstanding. Offline evaluation is therefore a fast filter that lets you reject bad changes cheaply and iterate without shipping, and it is not evidence that a change is good.

Clicks are evidence about position as much as about relevance

The tempting alternative is to skip judgements and use behaviour, since you have millions of clicks and no judges. The problem is that the top result gets clicked far more than the second whether or not it deserves to, because users scan from the top and stop early. So click-through rate at a position confounds relevance with prominence, and a ranking that reorders results changes clicks for reasons that have nothing to do with quality.

The consequence is a feedback loop, and it is worth naming because it is how a ranking model quietly degrades. Train on clicks, and the model learns to reproduce whatever the current ranking already promotes; the results it never showed get no clicks, so they look irrelevant, so they continue not to be shown. The model becomes confident and progressively less able to discover anything new.

Three techniques break it. Model the bias explicitly — estimate the probability that a result at each position was examined at all, and weight observed clicks by the inverse of that probability, so a click on a low position counts for more than a click on the first. Compare positions rather than counting them, which is what interleaving does: build one result list by alternating contributions from the two rankings, show it to the user, and attribute each click to whichever ranking supplied that result. Because both candidates are exposed to the same user in the same session, interleaving removes the variance between users that a conventional split test has to overcome, and it detects ranking differences with far less traffic. And inject a small amount of deliberate exploration, showing results the current ranking would not, so that you keep collecting evidence about documents your model has written off.

Interleaving's limit is what it can measure. It compares two orderings on clicks within a session, so it cannot tell you about anything that unfolds afterwards — whether the item was bought, returned, or watched to the end — and for that you need a conventional experiment with the real outcome as the metric.

The online experiment, and what it needs alongside the metric

Randomise by user rather than by query, so a single person does not see two different rankings and get an incoherent experience, and so that per-user outcomes are attributable. Then think about sensitivity before you run it: the effect a ranking change has on a business metric is usually small relative to its variance, and an experiment without enough traffic to detect a real effect returns "no difference", which is routinely misread as "safe to ship". Knowing what size of effect your traffic can actually detect is the part that stops you drawing a conclusion the data does not support.

Guardrails matter as much as the primary metric, because ranking changes fail in ways the metric being optimised is blind to. Watch the zero-result rate, since a change to matching can quietly stop retrieving anything for a class of queries. Watch abandonment and query reformulation, which are the clearest signals of a user who did not find what they came for. Watch latency, because a richer scoring stage that improves relevance and adds two hundred milliseconds may be a net loss. And watch the commercial outcome separately from engagement, since a change that increases clicks by promoting eye-catching irrelevant items is a regression wearing a metric's clothing.

The aggregate hides the queries that broke

The last discipline, and the one that separates a candidate who has shipped ranking changes from one who has read about evaluating them, is refusing to accept a single number. An improvement of a fraction of a point in mean nDCG is compatible with several very different realities: a small gain spread evenly, a large gain on head queries paired with serious losses in the tail, or no real change at all plus noise.

So decompose before deciding. Produce a per-query win-and-loss list ordered by magnitude, and read the twenty worst regressions individually — they are where you find that the change broke a whole query pattern, such as exact part numbers or multi-word brand names. Break the result out by segment: head, torso and tail, by query length, by whether the query names a category or a specific item, by locale. And check whether the improvement is concentrated in queries whose traffic is itself an artefact of the current ranking.

Then be sceptical of the first days of data. A visible change in result ordering produces a novelty response that fades, and an index rebuild or a caching change deployed alongside the ranking will confound the comparison. Running long enough to see the effect stabilise, and holding one segment back so you can confirm the gain persists, is what turns a promising number into a decision.

Offline judgements plus a rank-aware metric tell you cheaply whether a change is worth testing, an online experiment corrected for position bias tells you whether users agree, and neither is worth anything until you have looked at the twenty queries the change made worst.