hardConceptDesignMidSeniorStaff
Where does a cross-encoder reranker earn the latency it costs you?
5 min readUpdated 2026-08-15
Answer
Short answer
It earns it when your gold set shows recall at 50 is much better than recall at 5, because that gap is what reranking converts into answer quality. The cost is a model pass per candidate with nothing precomputable, so the latency budget divided by measured per-batch cost sets the candidate count.
Was this answer helpful?
Likely follow-ups
- Your reranker truncates passages at its input limit. Which chunks does that quietly penalise?
- How would you keep the reranker's benefit while halving its cost?
- Where does a late-interaction model sit between a bi-encoder and a cross-encoder?
- The reranker reorders the top five but the answer does not improve. What does that tell you?
Related questions
- Why is a recommender built as candidate generation followed by ranking rather than as one model?hardAlso on ranking and retrieval5 min
- Models take a million tokens of context now. Does that remove the need for retrieval?mediumAlso on retrieval and latency5 min
- Retrieval keeps returning plausible but unhelpful passages. How do you improve it without changing the model?hardAlso on retrieval and reranking6 min
- How do you decide whether a model should be served in batch, online or streaming?mediumAlso on latency4 min