Building a gold set from real questions, recall at k versus end-answer quality, groundedness and citation checks, and diagnosing which stage failed.
3 questions in RAG, Retrieval & Agents.
Source the questions from real users and from randomly sampled documents rather than from the team, label each with a quotable answer string anchored to a document rather than a chunk id so the set survives re-chunking, and hold part of it back so you are not tuning against your own measurement.
Recall at k is the primary one, since a passage that never reaches the prompt cannot be used; precision at k measures wasted window; MRR or NDCG measure ordering. Read them beside the groundedness of the answer, because high recall with bad answers points at generation or context ordering.
Walk three gates in order for that one request — was the supporting passage retrieved, did it survive into the rendered prompt, does the answer contradict it. Each gate blames a different stage, which is why the diagnosis depends on having logged both the retrieved chunk ids and the prompt as sent.