Loading...
Loading...
Browse 24 real-world technical and behavioral interview questions about Evaluation. Review scenarios, edge cases, and architectural best practices.
Serve context instead of history: popularity conditioned on the country, device and referrer you already have, then re-rank on the first two clicks of the session with a model that needs no user identity. Reserve slots for exploration, and segment every offline metric by history length, because a metric averaged over users with history cannot see this cohort at all.
Source the questions from real users and from randomly sampled documents rather than from the team, label each with a quotable answer string anchored to a document rather than a chunk id so the set survives re-chunking, and hold part of it back so you are not tuning against your own measurement.
Work backwards from the review capacity. Fix the alert budget the team can clear, evaluate precision and recall at exactly that budget on data at the real prevalence, define the label and its arrival delay before modelling, and never report accuracy, which a do-nothing model scores 99.95% on.
A judge is reliable for pairwise comparison against a concrete rubric and unreliable as an absolute scorer. It carries position, verbosity and self-preference biases and agrees with whatever the prompt implies, so validate it against human labels before trusting any number it gives you.
Almost certainly not: if fraud is 0.6% of transactions, a model predicting 'not fraud' for everything scores 99.4% and catches nothing. The metric has to follow from which error costs more, which makes it a business decision expressed in numbers rather than a modelling one.
Choose a classifier decision threshold from calibrated probability, false-positive cost, false-negative cost and operational capacity. The threshold changes when costs, prevalence, calibration or alert budget changes. It also connects cost sensitive learning to the point an interviewer is testing.
Almost always in retrieval rather than generation, and most often in chunking: if the answer was split across a boundary, or sits in a chunk whose surrounding context was stripped, no prompt can recover it. Evaluate the retrieval half separately, because it has a yes-or-no answer that needs no model to grade.
Data drift is a change in the distribution of the inputs; concept drift is a change in the relationship between inputs and the outcome. You can detect data drift from inputs alone, concept drift needs labels or a performance signal, and only concept drift necessarily invalidates what the model learned.
Walk three gates in order for that one request — was the supporting passage retrieved, did it survive into the rendered prompt, does the answer contradict it. Each gate blames a different stage, which is why the diagnosis depends on having logged both the retrieved chunk ids and the prompt as sent.
Harvest cases from real inputs and past incidents rather than inventing them, record for every case why it exists and what failure it guards, separate deterministic assertions from graded quality, and freeze a regression slice you never tune against.
Pin the model version, keep the prompt to stable intent and a schema, and record every model-specific workaround as dated, deletable debt. The durable artefacts are the output schema and the eval set - they are what tells you whether the new model is better before you ship it.
Report two rates separately: ungroundedness, meaning claims the provided sources do not support, and factual error, meaning claims wrong about the world. Measure per claim, publish the abstention rate beside them, and name the query set that produced the numbers. Use this hallucination answer to show the decision, trade-off, and evidence rather than a memorised definition.
Compare it against a genuinely strong prompted baseline on a held-out set drawn from real traffic, using the task metric your product cares about, and run a broad regression set alongside to catch capability lost elsewhere. Falling training loss is evidence that training worked, not that the product improved.
Scope a POC around the two or three technical claims genuinely in doubt, with written pass/fail exit criteria signed off by the person who will decide, a hard time box, and named customer resources committed before any work starts.
Read the failing queries first and sort them into causes, then apply the matching lever — rewriting for vague queries, filters for scope errors, smaller retrieval units for buried answers, lexical retrieval for literal terms, a reranker where recall sits below the cut — each with the metric that proves it worked.
Label latency caps your retraining cadence at roughly the label delay, forces evaluation to run on a lagging window that is always partially incomplete, and pushes you towards proxy labels that arrive sooner. Proxies buy speed at the price of optimising something adjacent to the outcome you care about. It also connects training data to the point an interviewer is testing.
Decide from the task. For grounded work over your own corpus, quality is set by what reaches the context, so retrieval dominates model size and a smaller model usually suffices. Keep the larger model for long multi-step reasoning, and treat routing as a second system to evaluate and monitor.
Inventory everywhere the model is coupled to behaviour, then use the eval set as your migration harness, shadow the candidate on live traffic to compare without user exposure, and re-measure cost and latency because both shift. Use the migration to delete instructions that only compensated for the old model.
ROC-AUC vs precision-recall is a metric choice for imbalanced classification. ROC-AUC measures ranking across all negatives, while PR curves expose alert precision and recall at the positive-class prevalence.
Treat the prompt as a versioned artefact under review, run your eval set in CI against a pinned model version, canary on a slice of traffic with the version recorded per request, and keep rollback a config change rather than a deploy. Pinning is what makes a regression attributable to your edit.
Decide which groups you are testing across and how you will obtain that attribute, choose one fairness definition and justify it - because the common definitions are mathematically incompatible and cannot all be satisfied - then report performance and error rates per group with sample sizes and intervals attached. It also connects bias testing to the point an interviewer is testing.
A model is calibrated when among all cases it scores 0.30, close to 30% are positive. Most classifiers emit a score that ranks well and is not a probability, so any decision built from expected value is wrong by an amount nobody measures. Fit isotonic or Platt scaling on a held-out set to repair it.
Whenever you can draw the flowchart. A fixed pipeline with a model at the two steps needing judgement is cheaper, testable and debuggable, and per-step error compounds across a planning loop. Reach for an agent when the tool sequence is open-ended and the environment verifies the work.
Recall at k is the primary one, since a passage that never reaches the prompt cannot be used; precision at k measures wasted window; MRR or NDCG measure ordering. Read them beside the groundedness of the answer, because high recall with bad answers points at generation or context ordering.