Building an offline eval set, LLM-as-judge and its failure modes, regression suites for prompts, and measuring hallucination honestly.
3 questions in Generative AI & Large Language Models.
A judge is reliable for pairwise comparison against a concrete rubric and unreliable as an absolute scorer. It carries position, verbosity and self-preference biases and agrees with whatever the prompt implies, so validate it against human labels before trusting any number it gives you.
Harvest cases from real inputs and past incidents rather than inventing them, record for every case why it exists and what failure it guards, separate deterministic assertions from graded quality, and freeze a regression slice you never tune against.
Report two rates separately: ungroundedness, meaning claims the provided sources do not support, and factual error, meaning claims wrong about the world. Measure per claim, publish the abstention rate beside them, and name the query set that produced the numbers.