hardConceptDesignMidSeniorStaff
You are using a model to grade another model's output. Where does that work, and where does it lie to you?
5 min readUpdated 2026-08-15
Answer
Short answer
A judge is reliable for pairwise comparison against a concrete rubric and unreliable as an absolute scorer.It carries position, verbosity and self-preference biases and agrees with whatever the prompt implies, so validate it against human labels before trusting any number it gives you.
Was this answer helpful?
Likely follow-ups
- How would you measure whether your judge agrees with your human raters, and what level would you accept?
- Your judge scores everything between 7 and 9. What is wrong and what do you change?
- When would you accept the same model family as both generator and judge?
- What happens if you start optimising prompts directly against the judge's score?
Related questions
- How do you build a gold set for a retrieval system?mediumAlso on evaluation5 min
- Your labels arrive weeks after the prediction. How does that shape what you can build?hardAlso on evaluation5 min
- How do you ship a change to a production prompt safely?mediumAlso on evaluation4 min
- How would you test a model for bias?hardAlso on evaluation6 min