mediumConceptDesignMidSeniorStaff
How do you build a gold set for a retrieval system?
5 min readUpdated 2026-08-15
Answer
Short answer
Source the questions from real users and from randomly sampled documents rather than from the team, label each with a quotable answer string anchored to a document rather than a chunk id so the set survives re-chunking, and hold part of it back so you are not tuning against your own measurement.
Was this answer helpful?
Likely follow-ups
- Two annotators disagree on whether a passage answers a question. How do you resolve it?
- How large does the set need to be before a five-point change in recall means anything?
- Your gold set was built a year ago and the corpus has moved on. What do you do with it?
- How would you bootstrap a gold set on day one, with no users and no query log?
Related questions
- Your RAG system cannot answer a question whose answer is definitely in the documents. Where is the fault?hardAlso on rag and retrieval4 min
- How do you choose a chunking strategy for a document corpus?mediumAlso on retrieval and rag4 min
- You are building a support assistant over a knowledge base that changes every week. RAG or fine-tuning, and how do you decide?mediumAlso on rag and evaluation5 min
- Retrieval keeps returning plausible but unhelpful passages. How do you improve it without changing the model?hardAlso on retrieval and evaluation6 min