Building an offline eval set, LLM-as-judge and its failure modes, regression suites for prompts, and measuring hallucination honestly.
4 questions in Generative AI & Large Language Models.
Price the error first, then shrink what the model decides: code applies policy and computes amounts, answers cite sources, and checks run against systems of record. Route a case to human review when its measured error rate makes review cheaper than the expected loss, and launch on an eval sized for that claim.
A judge is reliable for pairwise comparison against a concrete rubric and unreliable as an absolute scorer. It carries position, verbosity and self-preference biases and agrees with whatever the prompt implies, so validate it against human labels before trusting any number it gives you.
Harvest cases from real inputs and past incidents rather than inventing them, record for every case why it exists and what failure it guards, separate deterministic assertions from graded quality, and freeze a regression slice you never tune against.
Report two rates separately: ungroundedness, meaning claims the provided sources do not support, and factual error, meaning claims wrong about the world. Measure per claim, publish the abstention rate beside them, and name the query set that produced the numbers. Use this hallucination answer to show the decision, trade-off, and evidence rather than a memorised definition.