You are using a model to grade another model's output. Where does that work, and where does it lie to you?

What a judge is good for

Grading open-ended output at scale has no cheap alternative. Reference-based string metrics do not work when many different answers are correct, and human review does not run on every commit. A judge model closes that gap, and where it is used properly it is genuinely useful: comparing two candidate outputs for the same input, checking a rubric criterion that is concrete and checkable from the text, screening a large output set down to the cases a human should look at, and detecting obvious failures such as an answer that ignores the question or contradicts the supplied source.

The common thread is that the judge is asked something local and verifiable, ideally against material in front of it. It works far less well when asked to place a single output on an abstract quality scale from its own sense of what good looks like, because that is where its biases have nothing to push against.

The specific ways it misleads you

Four failure modes are well enough established to name, and naming them is what separates a real answer from a general caution.

Position bias. Presented with two candidates, judges systematically favour one of the positions — often the first — independently of content. If your pipeline always puts the new variant second, you have an offset baked into every comparison, and it is large enough to reverse conclusions on close calls.

Verbosity bias. Longer, more elaborate, more confident-sounding answers score higher at equal correctness. This is the most damaging one in practice, because it rewards exactly the behaviour that costs you output tokens and irritates users. A prompt change that makes answers 40 per cent longer will look like a quality win on judge scores and a loss in production.

Self-preference. A judge tends to prefer text produced by itself or by models of its own family, presumably because that text matches its own notion of well-formed output. Using one model to adjudicate between itself and a competitor is therefore not a neutral test, and it is the shape most vendor comparisons quietly take.

Sycophancy. The judge agrees with whatever the prompt leads it towards. Ask "is this answer correct?" and you get more yeses than asking "identify any errors in this answer". Mention that one candidate is the current production system and it wins more often. Any framing that reveals which output you hope will win contaminates the result, which is why judge prompts must be blind and neutrally worded.

To that list add the structural limit: the judge cannot check what it does not know. On specialist correctness — a clinical dosage, a tax rule, a claim about your own product's behaviour — a judge from the same pretraining tradition as the generator shares its misconceptions, so it confirms confident errors. There the only usable design is to give the judge the ground truth and ask it to compare, which turns it into a comparison tool rather than a source of knowledge.

Prefer comparison to scoring

Absolute numeric scoring is where judges are weakest. Ask for a mark out of ten and you get a distribution crushed into a narrow band near the top, unstable across runs, and not comparable between two evaluation sessions a month apart. It looks like a measurement and it is closer to a mood.

Pairwise preference is much more reliable, because "which of these two better satisfies this criterion" is a question with a local answer. Present A against B, then swap the order and ask again, and count only the cases where both orderings agree — the disagreements are your position-bias rate, which is itself worth tracking, and treating them as ties is more honest than picking one. Allow a genuine tie option, or the judge will manufacture a preference where none exists. Where you need something like a scalar, aggregate many pairwise results into a win rate against a fixed reference output rather than asking for a number directly.

If you must use a rubric with levels, make each level a description of observable properties rather than an adjective, require the judge to quote the evidence from the output before giving its verdict, and keep the criteria separate — one call per criterion, since a judge asked to weigh five dimensions at once collapses them into overall impression.

Validate the judge before you believe it

This is the step that is skipped, and everything else depends on it. A judge is a measuring instrument, and an uncalibrated instrument produces numbers with no known relationship to the quantity of interest.

Take a sample of a few hundred outputs, have humans label them against the same rubric the judge will use, and measure agreement — not raw percentage agreement alone, which is flattered by class imbalance, but a chance-corrected statistic. Compare it against the agreement your human raters achieve with each other, because that is the ceiling: a judge matching humans about as often as humans match each other is doing as well as the task permits, whereas one substantially below that is not usable for the decision you were about to make with it. Report that agreement figure alongside every judge-derived metric you publish, so a reader can discount appropriately.

Then keep the calibration alive. Re-validate when you change the judge model, the judge prompt or the rubric, and keep a small continuous human-labelled sample so drift shows up as a widening gap rather than as a surprise. Track judge cost too; a judge call is an LLM call, and judging every production output doubles your request volume.

Never optimise directly against it

The last trap is the one that turns a useful tool into a harmful one. Once a judge score becomes the objective — the number a prompt-tuning loop maximises, or the reward in a training run — you are optimising against a proxy with known, exploitable biases, and the optimiser will find them. What comes out is longer, more assertive, more flattering to the judge's preferences, and not better. Keep at least one human-reviewed set outside the loop as the arbiter, and treat a large gap between judge score and human score as evidence that the loop has learned the judge rather than the task.