Top 27 Evaluation Interview Questions
Browse 27 real-world technical and behavioral interview questions about Evaluation. Review scenarios, edge cases, and architectural best practices.
You are building a support assistant over a knowledge base that changes every week. RAG or fine-tuning, and how do you decide?
Use RAG for the knowledge, because articles change weekly and retrieval lets you edit, withdraw and cite them without retraining. Fine-tune only for behaviour retrieval cannot fix, such as tone, escalation judgement or reply format, and decide by auditing real failures and evaluating each fix on its own slice.
A wrong answer from the model would cost real money. How do you put it in the product anyway?
Price the error first, then shrink what the model decides: code applies policy and computes amounts, answers cite sources, and checks run against systems of record. Route a case to human review when its measured error rate makes review cheaper than the expected loss, and launch on an eval sized for that claim.
A team wants to launch a customer-facing LLM feature. As the accountable executive, what do you require before it ships?
Require AI governance in proportion to risk. Apply risk tiering to what the output does, who sees it and whether errors reverse. Every tier needs a named owner, a tested kill switch and fallback, data rules and monitoring; higher tiers add pre-agreed evals, adversarial testing, human review and legal sign-off.
The recommender is good for users with history and useless for the ones who just signed up. What do you serve them?
Serve context instead of history: popularity conditioned on the country, device and referrer you already have, then re-rank on the first two clicks of the session with a model that needs no user identity. Reserve slots for exploration, and segment every offline metric by history length, because a metric averaged over users with history cannot see this cohort at all.
How do you build a gold set for a retrieval system?
Source the questions from real users and from randomly sampled documents rather than from the team, label each with a quotable answer string anchored to a document rather than a chunk id so the set survives re-chunking, and hold part of it back so you are not tuning against your own measurement.
One transaction in two thousand is fraudulent and you have been asked to build the detector. How do you approach it?
Work backwards from the review capacity. Fix the alert budget the team can clear, evaluate precision and recall at exactly that budget on data at the real prevalence, define the label and its arrival delay before modelling, and never report accuracy, which a do-nothing model scores 99.95% on.
You are using a model to grade another model's output. Where does that work, and where does it lie to you?
A judge is reliable for pairwise comparison against a concrete rubric and unreliable as an absolute scorer. It carries position, verbosity and self-preference biases and agrees with whatever the prompt implies, so validate it against human labels before trusting any number it gives you.
Your fraud model is 99.4% accurate. Is that good?
Almost certainly not: if fraud is 0.6% of transactions, a model predicting 'not fraud' for everything scores 99.4% and catches nothing. The metric has to follow from which error costs more, which makes it a business decision expressed in numbers rather than a modelling one.
How do you choose the decision threshold for a classifier, and what makes it move?
Choose a classifier decision threshold from calibrated probability, false-positive cost, false-negative cost and operational capacity. The threshold changes when costs, prevalence, calibration or alert budget changes. It also connects cost sensitive learning to the point an interviewer is testing.
Your RAG system cannot answer a question whose answer is definitely in the documents. Where is the fault?
Almost always in retrieval rather than generation, and most often in chunking: if the answer was split across a boundary, or sits in a chunk whose surrounding context was stripped, no prompt can recover it. Evaluate the retrieval half separately, because it has a yes-or-no answer that needs no model to grade.
What is the difference between data drift and concept drift, and does it change what you do?
Data drift is a change in the distribution of the inputs; concept drift is a change in the relationship between inputs and the outcome. You can detect data drift from inputs alone, concept drift needs labels or a performance signal, and only concept drift necessarily invalidates what the model learned.
A user reports a wrong answer. How do you work out whether retrieval or generation failed?
Walk three gates in order for that one request — was the supporting passage retrieved, did it survive into the rendered prompt, does the answer contradict it. Each gate blames a different stage, which is why the diagnosis depends on having logged both the retrieved chunk ids and the prompt as sent.
You are shipping an LLM feature and there is no evaluation set. How do you build one?
Harvest cases from real inputs and past incidents rather than inventing them, record for every case why it exists and what failure it guards, separate deterministic assertions from graded quality, and freeze a regression slice you never tune against.
How do you write prompts that survive a model upgrade?
Pin the model version, keep the prompt to stable intent and a schema, and record every model-specific workaround as dated, deletable debt. The durable artefacts are the output schema and the eval set - they are what tells you whether the new model is better before you ship it.
How do you measure hallucination in a way you would put in front of a customer?
Report two rates separately: ungroundedness, meaning claims the provided sources do not support, and factual error, meaning claims wrong about the world. Measure per claim, publish the abstention rate beside them, and name the query set that produced the numbers. Use this hallucination answer to show the decision, trade-off, and evidence rather than a memorised definition.
You have fine-tuned a model for a task. How do you prove it helped?
Compare it against a genuinely strong prompted baseline on a held-out set drawn from real traffic, using the task metric your product cares about, and run a broad regression set alongside to catch capability lost elsewhere. Falling training loss is evidence that training worked, not that the product improved.
How do you scope a proof of concept so that it closes the deal?
Scope a POC around the two or three technical claims genuinely in doubt, with written pass/fail exit criteria signed off by the person who will decide, a hard time box, and named customer resources committed before any work starts.
Retrieval keeps returning plausible but unhelpful passages. How do you improve it without changing the model?
Read the failing queries first and sort them into causes, then apply the matching lever — rewriting for vague queries, filters for scope errors, smaller retrieval units for buried answers, lexical retrieval for literal terms, a reranker where recall sits below the cut — each with the metric that proves it worked.
Your labels arrive weeks after the prediction. How does that shape what you can build?
Label latency caps your retraining cadence at roughly the label delay, forces evaluation to run on a lagging window that is always partially incomplete, and pushes you towards proxy labels that arrive sooner. Proxies buy speed at the price of optimising something adjacent to the outcome you care about. It also connects training data to the point an interviewer is testing.
Do you use one large model, or a smaller model with retrieval? How do you decide?
Decide from the task. For grounded work over your own corpus, quality is set by what reaches the context, so retrieval dominates model size and a smaller model usually suffices. Keep the larger model for long multi-step reasoning, and treat routing as a second system to evaluate and monitor.
Your provider announces the model you depend on retires in six weeks. Walk me through what you do.
Inventory everywhere the model is coupled to behaviour, then use the eval set as your migration harness, shadow the candidate on live traffic to compare without user exposure, and re-measure cost and latency because both shift. Use the migration to delete instructions that only compensated for the old model.
When does ROC-AUC mislead you, and when does a precision-recall curve mislead you instead?
ROC-AUC vs precision-recall is a metric choice for imbalanced classification. ROC-AUC measures ranking across all negatives, while PR curves expose alert precision and recall at the positive-class prevalence.
How do you ship a change to a production prompt safely?
Treat the prompt as a versioned artefact under review, run your eval set in CI against a pinned model version, canary on a slice of traffic with the version recorded per request, and keep rollback a config change rather than a deploy. Pinning is what makes a regression attributable to your edit.
How would you test a model for bias?
Decide which groups you are testing across and how you will obtain that attribute, choose one fairness definition and justify it - because the common definitions are mathematically incompatible and cannot all be satisfied - then report performance and error rates per group with sample sizes and intervals attached. It also connects bias testing to the point an interviewer is testing.
What does it mean for a model to be calibrated, and why is an uncalibrated probability dangerous?
A model is calibrated when among all cases it scores 0.30, close to 30% are positive. Most classifiers emit a score that ranks well and is not a probability, so any decision built from expected value is wrong by an amount nobody measures. Fit isotonic or Platt scaling on a held-out set to repair it.
When is an agent the wrong shape for a problem?
Whenever you can draw the flowchart. A fixed pipeline with a model at the two steps needing judgement is cheaper, testable and debuggable, and per-step error compounds across a planning loop. Reach for an agent when the tool sequence is open-ended and the environment verifies the work.
Which metrics tell you whether retrieval is working?
Recall at k is the primary one, since a passage that never reaches the prompt cannot be used; precision at k measures wasted window; MRR or NDCG measure ordering. Read them beside the groundedness of the answer, because high recall with bad answers points at generation or context ordering.