hardDesignScenarioMidSeniorStaff
You are shipping an LLM feature and there is no evaluation set. How do you build one?
5 min readUpdated 2026-08-15
Answer
Short answer
Harvest cases from real inputs and past incidents rather than inventing them, record for every case why it exists and what failure it guards, separate deterministic assertions from graded quality, and freeze a regression slice you never tune against.
Was this answer helpful?
Likely follow-ups
- How do you get a first set before the feature has any traffic at all?
- You have 400 cases and a prompt change takes an hour to evaluate. What do you do?
- Would you generate synthetic cases with a model, and what does that inherit?
- How do you keep the set from silently drifting out of date as the product changes?
Related questions
- You have fine-tuned a model for a task. How do you prove it helped?hardAlso on evaluation and regression-testing5 min
- You are building a support assistant over a knowledge base that changes every week. RAG or fine-tuning, and how do you decide?mediumAlso on evaluation5 min
- The recommender is good for users with history and useless for the ones who just signed up. What do you serve them?hardAlso on evaluation7 min
- Your API suite runs on every commit against a shared database. How do you keep the tests from corrupting each other's data?hardAlso on test-data6 min