How do you build a gold set for a retrieval system?

The set decides what you can learn, so its bias is the whole risk

Every retrieval decision you make afterwards is judged against this set, which means a bias in the set becomes a bias in the system, invisibly and permanently. The number goes up, the users keep complaining, and nobody can reconcile the two. So the interesting work is not the labelling format but where the questions come from.

The default source is the team, and it is the wrong one. People who built the system write questions in the corpus's own vocabulary because they have read the corpus, they write well-formed questions because they know what a retriever likes, and they unconsciously write questions they have seen work. The result is a set clustered around the behaviour that already exists, which is exactly the region where measurement tells you nothing. It also systematically omits the categories that break things: bare identifiers, one-word queries, follow-ups that depend on a previous turn, questions with a date in them, and questions the corpus cannot answer at all.

Where the questions actually come from

Real user questions, wherever they already exist. A support ticket queue, the search log of the intranet you are replacing, the questions sales engineers get asked, the questions in a help-desk chat channel. These are free, they are in the users' own words, and they carry the distribution you have to serve rather than the one you would prefer. If the product is live, sample from its own query log stratified by frequency so you get both the head and the long tail, because the tail is where retrieval fails and the head is what users notice.

Known-answer documents, sampled at random. Take a document from the corpus by random selection rather than by choosing one, read it, and write the questions a user would plausibly ask that this document answers. Random selection is what breaks the team's bias: you end up writing questions about the awkward appendix, the badly formatted spreadsheet and the deck with three words per slide, rather than about the well-structured policy page. Recording that the answer lives in this document is free, because you started from the document.

Questions with no answer in the corpus. A gold set of only answerable questions cannot measure the behaviour that matters most for trust, which is declining to answer. Include a deliberate portion where the correct outcome is that retrieval returns nothing useful and the system says so.

Combine the sources and stratify the result, so you can report per class: literal or identifier queries, paraphrased queries, multi-part queries needing two documents, time-scoped queries, and unanswerable ones. A single aggregate number over an unstratified set hides the case where an improvement raised the average and destroyed one class.

Label the answer, not the chunk

The labelling decision that saves the most future work is what a label points at. The temptation is to record the chunk identifier that was returned when you tried the query, and that binds the gold set to the current chunking strategy. Re-chunk the corpus — which you will, since chunk size is a parameter you intend to test — and every label is stale, so the fixture that exists to compare strategies is destroyed by the first strategy change.

Anchor the label instead to the document identifier plus a verbatim answer string quoted from it.

- question: "How long do I have to return a priority delivery item?"
  doc_id: "returns-policy-2026"
  answer_quote: "returned within 14 days of delivery"
  class: paraphrase
  answerable: true

The quoted string makes scoring mechanical and chunking-independent: a retrieved chunk is a hit if it contains the string, whatever the boundaries are. It is also robust to reindexing, embedding model changes and index type changes, which is exactly the set of things you want to compare. Where a question genuinely needs two documents, record both, and score it only when both are retrieved, because partial credit on multi-hop questions flatters a retriever that reliably finds the easy half.

Graded relevance is worth the effort only if you intend to use a metric that consumes it. Binary judgements are enough for recall at k and cheaper to obtain consistently; graded labels are needed for NDCG and they are where annotator disagreement concentrates. If two annotators disagree, that usually means the question is ambiguous, and an ambiguous question in a gold set is noise rather than difficulty, so the right resolution is often to rewrite or discard it. Measure agreement on a sample early; a set nobody can label consistently cannot be a fixture.

Size, freezing and the set you do not look at

The set needs to be large enough that the change you care about is distinguishable from sampling noise. That is a matter of arithmetic rather than a round number: with fifty questions, each one is two percentage points of recall, so a two-point improvement is one question changing its mind and means nothing. Start with a set you can build in a day and grow it, using per-class counts rather than a global total, because a class with four questions in it is not measured at all.

Then treat it as a fixture. Version it in the repository next to the code, review changes to it as you would review code, and never edit a label because a change you liked failed on it. That last one is the most common way a gold set dies. To make it harder to do accidentally, split it: a development portion you evaluate against freely while iterating, and a held-back portion you run rarely, at release decision points. Tuning against the same questions you judge with produces a system fitted to fifty examples, and it will look excellent right up to the point where someone else uses it.

Finally, keep it alive. Add every reported failure as a new case once it is diagnosed, which turns the set into a regression suite that grows in exactly the places the system is weak, and re-verify labels when the corpus changes underneath them, since a document that has been rewritten no longer contains the quoted string.

The team's questions measure the system you already have. Real user questions and randomly sampled documents are what tell you about the queries it fails on, and those are the only ones worth measuring.