Retrieval keeps returning plausible but unhelpful passages. How do you improve it without changing the model?

Read the failures before choosing a lever

"Plausible but unhelpful" is a symptom with several causes, and the levers available treat different ones. So the first move is not a change to the pipeline; it is to collect thirty or forty failing queries, look at what came back for each, and label the cause. This takes an afternoon and it is the difference between fixing the problem and cycling through techniques.

The labels that matter are few. The passage was never in the candidate set — a retrieval miss, and no reranking can help. It was in the candidate set but ranked below the cut — a ranking problem, which is what a reranker is for. The query was underspecified, so the retriever answered a vaguer question than the user meant. The wrong version or scope came back: last year's rate card, another region's policy, a superseded document. Or the passage was retrieved and was genuinely useless, because the corpus does not contain the answer, which is a content problem masquerading as a retrieval one.

flowchart TD
    A[Failing query] --> B{What came back}
    B -->|Passage absent entirely| C{Is the query literal or vague}
    B -->|Present but ranked below the cut| D[Rerank the wide candidate set]
    B -->|Wrong version region or date| G[Filter on date version and tenant]
    B -->|Right section but answer buried| H[Smaller retrieval unit with parent context]
    C -->|Literal term or identifier| E[Add lexical retrieval and fuse]
    C -->|Vague or multi-part| F[Rewrite or decompose the query]

The branch worth dwelling on is the first one, because everything downstream depends on it and it is answerable mechanically: search the retrieved candidate texts for the known answer string. Teams that skip it spend weeks tuning a reranker over a candidate set that never contained the passage.

Rewriting the query, and how to know it helped

Users write short, context-dependent, sometimes conversational queries, and the retriever sees only the string. Three rewrites earn their place. Contextualisation resolves a follow-up against the conversation, turning "what about for annual plans" into a standalone question, and without it multi-turn retrieval degrades badly because the pronouns carry the meaning. Decomposition splits a query with two requirements into separate retrievals whose results are merged, because a single vector cannot be near two unrelated passages at once. Expansion adds the vocabulary the corpus uses rather than the vocabulary the user used, which can be done from a synonym list, or by having a model draft a hypothetical answer and embedding that instead of the question, so the query text resembles the documents you are searching.

Every one of these is a model call in the request path, so each costs latency and can fail. The measurement is retrieval recall at k on the gold set with rewriting on and off, segmented by query type. Segmentation is essential here because rewriting reliably helps vague queries and reliably harms precise ones — an expansion applied to a query that is a single part number adds noise around a term that was already exact, and the fused average can improve while your best-served query class gets worse. If you cannot segment, at least check that no gold question regressed from a hit to a miss.

Filters do properly what retrieval does approximately

When the failure is scope — the wrong year, the wrong region, a superseded version, another tenant's document — no amount of ranking work is the right answer, because the constraint is a fact rather than a similarity. Embeddings do not encode magnitude or recency, so a question about the current policy retrieves passages about policies and the current one wins only by luck.

If you captured the metadata at ingestion, this is a filter applied before or during the search: effective date range, document status, region, tenant. The improvement is usually the largest single one available and it is often mistaken for a model problem for months. Measure it as precision at k on the affected slice, and additionally count how often a superseded document appears in a cited answer, because that count is what a stakeholder actually complains about.

The caveat is what a very selective filter does to an approximate index. Filtering results after the search means a query whose filter matches a small fraction of the corpus can return nothing at all, because the nearest hundred neighbours were all excluded. Filtering during the graph or cell traversal avoids that, and where the filter is extremely selective an exact scan of the filtered subset is both simpler and better.

Chunking and hybrid retrieval, in the order their cost implies

Two levers remain, and they differ sharply in what they cost to try. Adding lexical retrieval and fusing the rankings needs an index built alongside the existing one and no change to embeddings, and it directly addresses the literal-term class. Changing the retrieval unit — smaller focused chunks with a larger parent section passed to the model once a chunk is selected, or a header repeated into each row of a table — requires re-chunking and re-embedding the corpus, which is the most expensive item on this list.

That ordering argument is worth making explicitly, because it is the part that shows judgement. Try the levers in ascending order of cost and descending order of the evidence you have for them: filters first if the failure labels say scope, then hybrid if they say literal terms, then reranking if the recall gap between a wide and a narrow cut is large, then rewriting, and re-chunking last unless the labels overwhelmingly point at it. Reranking in particular is attractive because it needs no reindexing and can be removed by a flag, which makes it a safe experiment.

One change at a time, against a fixture

The failure mode of this whole exercise is shipping four improvements in one release and being unable to say which helped, which is how a pipeline accumulates a rewriting step nobody can justify and a fusion weight nobody dares to touch. Fix the gold set first, take a baseline, then change one thing and re-measure. Keep recall at k and end-answer quality both in the report, because they can move in opposite directions: a reranker that tightens the top five can drop a supporting passage the answer needed, and an expansion that raises recall can fill the window with near-duplicates that crowd out the one useful passage.

Finally, be willing to conclude that the corpus is the problem. If the labelled failures say the answer is not in any indexed document, the fix is ingestion coverage or content authoring, and every hour spent on the retriever is wasted. Saying so is a stronger answer than producing another technique.

Label the failing queries before touching the pipeline. Each lever here treats one specific cause, and applying them in a batch guarantees you will not learn which of them was the one that mattered.