Where does a cross-encoder reranker earn the latency it costs you?

Why the cost is per candidate and cannot be amortised

A bi-encoder — the ordinary embedding model behind vector search — encodes the query and each passage independently. That independence is the whole reason vector search is fast: passage vectors are computed once at ingestion and stored, and a query costs one encode plus a nearest-neighbour lookup. The price of the independence is that the model never sees the query and the passage together, so it cannot notice that the passage answers this particular question rather than merely discussing the same subject.

A cross-encoder takes the query and one passage concatenated as a single input and attends across both, producing a relevance score directly. That joint attention is what lets it distinguish a claim from its negation, notice that the number in the passage is not the number in the query, and prefer the paragraph containing the answer over the heading that restates the question. It is also why nothing can be precomputed: the score depends on the pair, so there is no passage-side artefact to cache, and a new query means a fresh model pass for every candidate you want scored.

That asymmetry sets the architecture. You cannot run a cross-encoder over the corpus, so it runs over a shortlist that a cheap retriever produced. Retrieve wide, rerank narrow.

Deriving the candidate count from the budget

The candidate count is where candidates either sound thoughtful or sound like they read a tutorial. Do the arithmetic from stated assumptions rather than naming a number.

Reranking cost is roughly linear in candidates multiplied by the tokens per candidate, with batching turning some of that into parallel work on an accelerator, so the practical unit is cost per batch. Suppose you measure your reranker and find it scores a batch of 32 passages of your typical length in 40 ms on your hardware. Then 64 candidates cost two batches, and 128 cost four, and the question becomes how many batch-times your budget can absorb. If the end-to-end answer budget is 3 seconds and generation reliably takes 2, you have around 1 second for everything before it, of which retrieval itself takes some, so a few batches fit and forty do not. The number falls out; you do not choose it.

Two things make that arithmetic honest. Measure on your own passage lengths, because cost scales with input tokens and a corpus of long chunks is a different problem from one of short rows. And measure at the tail rather than the mean, because a p99 that busts the budget is a timeout for a real user, and rerankers have very stable per-batch cost, which means the tail comes from queue depth under concurrency rather than from the model.

The gap that tells you whether it will help at all

Before building any of it, there is a measurement that predicts the benefit. On your gold set, compute retrieval recall at the wide cut you would rerank — say 50 — and at the narrow cut you would pass to the model, say 5. The difference between those two numbers is the entire opportunity: it is the set of questions where the right passage was retrieved but not highly enough to reach the prompt, and reranking is the mechanism for promoting it.

wide = recall_at_k(gold, k=50)
narrow = recall_at_k(gold, k=5)
# The headroom a reranker can capture. Small headroom, no reranker.
print(f"reranker headroom: {wide - narrow:.2%}")

If recall at 5 is already close to recall at 50, a perfect reranker changes nothing, because the ordering within the top five is already right and the misses are absent from the candidate set entirely. In that case your problem is upstream — chunking, hybrid retrieval, the query itself — and adding a reranker buys latency and an extra service for no gain. If the gap is wide, reranking is usually the cheapest quality improvement available, because it needs no reindexing and no change to ingestion.

The gap also tells you how wide to retrieve. Recall at 200 that is barely better than recall at 50 means candidates beyond 50 contain almost nothing worth promoting, so widening the funnel spends reranker time on passages that cannot help.

The truncation nobody notices

A cross-encoder has an input limit, and query plus passage must fit inside it. When a passage exceeds the remaining space it is truncated, and the score is computed on the head of the passage rather than on the passage. This penalises exactly the chunks whose relevant sentence sits late in the text — a long section whose answer is in its final paragraph, a table whose relevant row is near the bottom — and it does so silently, producing a low score that looks like a judgement about relevance.

The interaction with chunking matters here. If your chunks are comfortably inside the reranker's limit, this never bites; if your chunk strategy produced large sections, the reranker is scoring a prefix, and the fix is either smaller retrieval units or scoring passage windows and taking the maximum. Either way it is a thing to check rather than to assume, because it is invisible in an evaluation that only reports an aggregate score.

When to leave it out

Reranking is not free beyond latency. It is another model to host or another vendor call in the request path, another failure mode, and another version to pin. So there are cases to decline: a small headroom as measured above; a latency budget already consumed by generation; an interactive product where perceived speed is the feature. There are also middle options worth naming. A late-interaction retriever keeps per-token passage representations and scores with a cheap interaction at query time, which sits between the two extremes in both cost and quality. A small distilled reranker over a wider funnel sometimes beats a large one over a narrow funnel at the same cost. And if the reranker is a hosted call, the timeout must degrade to the unreranked order rather than failing the request, because the unreranked order was good enough to be your baseline.

The reranker's value is exactly the recall you have between the wide cut and the narrow one. Measure that gap first, then let the latency budget divided by measured per-batch cost decide how many candidates you can afford.