Why is a recommender built as candidate generation followed by ranking rather than as one model?
5 min readUpdated 2026-08-15
0
Answer
Short answer
Because you cannot score a million-item catalogue per request inside a few tens of milliseconds. Retrieval reduces millions to hundreds with a cheap model judged on recall, then a far more expensive ranker orders those hundreds and is judged on the ordering it produces at the top of the list.
Was this answer helpful?
Likely follow-ups
→How would you tell whether retrieval or ranking is your bottleneck?
→Why can the ranker use features the retrieval stage cannot?
→What breaks if you train the ranker only on items retrieval already surfaces?
→Where would you insert a business rule that must never be violated, and why not earlier?
Work the arithmetic out loud, because the architecture is a consequence of it
rather than a convention. Suppose the catalogue holds two million items and the
recommendation must be returned inside eighty milliseconds, of which perhaps fifty
are yours after the network and the surrounding page.
The model you would actually like to use is a cross-feature ranker: it sees the
user, the item, the context, and the interactions between them, and it costs
something in the region of tens of microseconds per item to evaluate. Two million
items at even ten microseconds each is twenty seconds of compute per request. It
is not close, and no amount of hardware closes a gap of that shape at the query
rate a consumer product runs at.
So you spend the budget unevenly. A cheap stage looks at everything and keeps a
few hundred plausible items. An expensive stage looks only at those few hundred.
Two million cheap operations plus a few hundred expensive ones fits; two million
expensive ones never will.
Loading diagram…
The interesting part is the narrowing between the first two boxes: everything the
ranker can ever return has already been decided there.
What the retrieval stage can afford to be
Retrieval has to touch the whole catalogue, so its per-item cost must be nearly
zero. The standard construction is a two-tower model: one tower embeds the user
and context, the other embeds the item, and relevance is a dot product between the
two vectors. The critical property is that the item tower does not depend on the
user, so every item embedding is computed offline and loaded into an approximate
nearest-neighbour index. At request time you embed the user once and the index
returns the nearest few hundred item vectors in single-digit milliseconds.
That property is also the constraint. Because the two towers never meet before the
dot product, retrieval cannot represent an interaction between a user feature and
an item feature — no "this user buys this brand only when it is discounted",
because that requires the model to see both sides together. It also cannot use
anything that only exists at request time in combination with the item, such as
the item's price after a user-specific promotion.
Retrieval is rarely one model in production. You run several sources in parallel
and merge them: the embedding index, an index of items similar to what the user
just viewed, recent popularity within their segment, items from a subscribed
creator, an explicit exploration source. Each covers a case the others miss, and
each can be turned off independently when it breaks.
What the ranker can afford to be
With a few hundred candidates the arithmetic inverts and you can spend real
compute per item. The ranker takes the full cross product of features — user
history, item attributes, the pair's co-occurrence statistics, request context,
freshness, price, position — and typically predicts several targets at once, since
the objective is usually a weighted combination of click, dwell, conversion and a
negative term for hiding or reporting the item.
Two consequences follow. The ranker is where the product's value function actually
lives, so a change to what the business wants is usually a change to the ranker's
objective weights rather than to retrieval. And the ranker is where model
complexity is affordable, which is why the deep-learning investment in
recommenders concentrates here.
Business rules and diversity belong after ranking, not before. A hard constraint —
suppress out-of-stock items, honour a user's block list, respect regional
licensing — is applied as a filter over the ranked list because that is the only
place you can guarantee it holds for what is displayed. Diversity and
deduplication also need the ordered list, since the question is whether the
second item is too similar to the first, which is not answerable per item.
The two stages are graded on different things
This is the part interviews probe hardest, and getting it wrong signals that the
split was memorised rather than understood.
Retrieval is judged on recall at k: of the items the user would have engaged
with, what fraction appeared in the candidate set. Precision is not its job and
ordering within the candidate set is not its job either, because the ranker will
reorder everything. A retrieval stage that returns the right item in position 400
has done its work perfectly.
Ranking is judged on ordering quality at the top of the list — NDCG, MRR, or
click-through and conversion at the positions actually displayed. Recall is
meaningless here because the candidate set is the universe it was given.
The relationship between the two is the number worth tracking. Retrieval recall is
a hard ceiling on the whole system: an item that never enters the candidate set can
never be recommended, and no ranker improvement can recover it. So measure it
directly by taking the items users engaged with and checking what share the
retrieval stage would have surfaced. If that figure is 0.6, then forty per cent of
your achievable value is unreachable no matter how good the ranker gets, and
another week of ranker work is the wrong week.
The feedback loop the split creates
The failure mode specific to this architecture is that the ranker's training data
is drawn entirely from items retrieval chose to surface. Items the retrieval stage
has never returned generate no impressions, so they generate no labels, so the
ranker never learns anything about them — and if the ranker's own scores are used
to update the retrieval model or to define its training negatives, the whole system
converges onto a narrowing slice of the catalogue while every offline metric
improves.
The defences are structural rather than clever. Reserve a slot for exploration so
some fraction of impressions come from candidates the system is uncertain about,
and log which source produced every impression so you can attribute value to each
retrieval stream. Sample negatives for the ranker from the catalogue at large
rather than only from the impressions log, so it has seen items outside the
candidate distribution. And when you evaluate offline, remember you are evaluating
on logged data whose composition your previous model chose, which is why the
offline number and the live number diverge in a predictable direction.