Why is a recommender built as candidate generation followed by ranking rather than as one model?

Derive it from the budget

Work the arithmetic out loud, because the architecture is a consequence of it rather than a convention. Suppose the catalogue holds two million items and the recommendation must be returned inside eighty milliseconds, of which perhaps fifty are yours after the network and the surrounding page.

The model you would actually like to use is a cross-feature ranker: it sees the user, the item, the context, and the interactions between them, and it costs something in the region of tens of microseconds per item to evaluate. Two million items at even ten microseconds each is twenty seconds of compute per request. It is not close, and no amount of hardware closes a gap of that shape at the query rate a consumer product runs at.

So you spend the budget unevenly. A cheap stage looks at everything and keeps a few hundred plausible items. An expensive stage looks only at those few hundred. Two million cheap operations plus a few hundred expensive ones fits; two million expensive ones never will.

flowchart LR
  C[Catalogue<br/>~2M items] --> R[Retrieval<br/>ANN plus heuristics]
  R --> P[Candidates<br/>~500]
  P --> K[Ranker<br/>rich cross features]
  K --> B[Business rules<br/>and diversity]
  B --> S[Final slate<br/>~20 shown]

The interesting part is the narrowing between the first two boxes: everything the ranker can ever return has already been decided there.

What the retrieval stage can afford to be

Retrieval has to touch the whole catalogue, so its per-item cost must be nearly zero. The standard construction is a two-tower model: one tower embeds the user and context, the other embeds the item, and relevance is a dot product between the two vectors. The critical property is that the item tower does not depend on the user, so every item embedding is computed offline and loaded into an approximate nearest-neighbour index. At request time you embed the user once and the index returns the nearest few hundred item vectors in single-digit milliseconds.

That property is also the constraint. Because the two towers never meet before the dot product, retrieval cannot represent an interaction between a user feature and an item feature — no "this user buys this brand only when it is discounted", because that requires the model to see both sides together. It also cannot use anything that only exists at request time in combination with the item, such as the item's price after a user-specific promotion.

Retrieval is rarely one model in production. You run several sources in parallel and merge them: the embedding index, an index of items similar to what the user just viewed, recent popularity within their segment, items from a subscribed creator, an explicit exploration source. Each covers a case the others miss, and each can be turned off independently when it breaks.

What the ranker can afford to be

With a few hundred candidates the arithmetic inverts and you can spend real compute per item. The ranker takes the full cross product of features — user history, item attributes, the pair's co-occurrence statistics, request context, freshness, price, position — and typically predicts several targets at once, since the objective is usually a weighted combination of click, dwell, conversion and a negative term for hiding or reporting the item.

Two consequences follow. The ranker is where the product's value function actually lives, so a change to what the business wants is usually a change to the ranker's objective weights rather than to retrieval. And the ranker is where model complexity is affordable, which is why the deep-learning investment in recommenders concentrates here.

Business rules and diversity belong after ranking, not before. A hard constraint — suppress out-of-stock items, honour a user's block list, respect regional licensing — is applied as a filter over the ranked list because that is the only place you can guarantee it holds for what is displayed. Diversity and deduplication also need the ordered list, since the question is whether the second item is too similar to the first, which is not answerable per item.

The two stages are graded on different things

This is the part interviews probe hardest, and getting it wrong signals that the split was memorised rather than understood.

Retrieval is judged on recall at k: of the items the user would have engaged with, what fraction appeared in the candidate set. Precision is not its job and ordering within the candidate set is not its job either, because the ranker will reorder everything. A retrieval stage that returns the right item in position 400 has done its work perfectly.

Ranking is judged on ordering quality at the top of the list — NDCG, MRR, or click-through and conversion at the positions actually displayed. Recall is meaningless here because the candidate set is the universe it was given.

The relationship between the two is the number worth tracking. Retrieval recall is a hard ceiling on the whole system: an item that never enters the candidate set can never be recommended, and no ranker improvement can recover it. So measure it directly by taking the items users engaged with and checking what share the retrieval stage would have surfaced. If that figure is 0.6, then forty per cent of your achievable value is unreachable no matter how good the ranker gets, and another week of ranker work is the wrong week.

The feedback loop the split creates

The failure mode specific to this architecture is that the ranker's training data is drawn entirely from items retrieval chose to surface. Items the retrieval stage has never returned generate no impressions, so they generate no labels, so the ranker never learns anything about them — and if the ranker's own scores are used to update the retrieval model or to define its training negatives, the whole system converges onto a narrowing slice of the catalogue while every offline metric improves.

The defences are structural rather than clever. Reserve a slot for exploration so some fraction of impressions come from candidates the system is uncertain about, and log which source produced every impression so you can attribute value to each retrieval stream. Sample negatives for the ranker from the catalogue at large rather than only from the impressions log, so it has seen items outside the candidate distribution. And when you evaluate offline, remember you are evaluating on logged data whose composition your previous model chose, which is why the offline number and the live number diverge in a predictable direction.