Why dense retrieval alone misses exact terms, combining BM25 with vectors, fusion strategies, and where a cross-encoder reranker earns its latency.
3 questions in RAG, Retrieval & Agents.
Read the failing queries first and sort them into causes, then apply the matching lever — rewriting for vague queries, filters for scope errors, smaller retrieval units for buried answers, lexical retrieval for literal terms, a reranker where recall sits below the cut — each with the metric that proves it worked.
It earns it when your gold set shows recall at 50 is much better than recall at 5, because that gap is what reranking converts into answer quality. The cost is a model pass per candidate with nothing precomputable, so the latency budget divided by measured per-batch cost sets the candidate count.
Dense retrieval handles paraphrase and misses literal tokens, while BM25 does the opposite, so you run both and merge the candidate lists. Merging by rank with reciprocal rank fusion is the robust default, because the two engines produce scores on scales that cannot be normalised reliably per query.