Top 12 Metrics Interview Questions
Browse 12 real-world technical and behavioral interview questions about Metrics. Review scenarios, edge cases, and architectural best practices.
Engineers added a user_id label to a few metrics and the observability bill went up tenfold. Explain what actually happened, and how you would fix it without losing the ability to debug.
Every distinct combination of label values is a separate time series with its own index entry, so an unbounded label multiplies series count by the number of users rather than adding one dimension. Metrics are the wrong tool for per-entity questions: keep labels bounded, move per-user debugging to exemplar-linked traces, and guard against the next unbounded label.
Your A/B test was configured for a 50/50 split but the data shows 53/47 across two million users. The result is significant and positive. What do you do?
Treat it as a broken experiment, not a rounding issue. At two million users a 53/47 split is astronomically unlikely by chance, so something removed users non-randomly - and that something almost certainly correlated with the outcome. The result cannot be trusted in either direction, so find the mechanism, fix it and rerun rather than adjusting the numbers.
Your headline metric went up and the business got worse. How does that happen, and how do you catch it?
Ratios improve when the denominator shrinks and averages improve when the mix shifts, so a rate can rise while every underlying cohort worsens; you catch it by pairing every rate with its absolute counts, decomposing by segment, and defining guardrails before launch.
Your fraud model is 99.4% accurate. Is that good?
Almost certainly not: if fraud is 0.6% of transactions, a model predicting 'not fraud' for everything scores 99.4% and catches nothing. The metric has to follow from which error costs more, which makes it a business decision expressed in numbers rather than a modelling one.
Daily active users dropped 15% week over week. How do you diagnose it?
Rule out instrumentation before behaviour, then segment along the dimensions that isolate a cause — platform, geography, cohort, and acquisition channel — to distinguish a broad drop from a concentrated one, since those have entirely different explanations.
How would you improve a product you use every day?
It is a structure test, not a creativity test: state the product's goal and how it earns money, name one segment and one unmet need, generate several options before choosing, then commit to a metric and the risk that would prove you wrong.
Leadership has asked for 20 per cent growth in activations next quarter. How do you decide whether that is the right target?
Build the number from a baseline and named mechanisms rather than accepting it, because a target with no route attached is a wish that will be met by lowering the definition. Decompose the growth into drivers, state what each is worth, and say plainly which portion of the gap has no plan behind it.
Latency on a checkout endpoint has tripled and nobody knows why. What should your metrics, logs and traces already have told you before anyone opens an editor?
Metrics tell you something is wrong and when it started, traces tell you which hop owns the latency, logs tell you what happened to one request. Cardinality is what makes metrics expensive, sampling is what makes traces affordable, and the goal is answering new questions from data already held.
Your DevRel programme reports growing impressions, follower counts and packed conference sessions, but finance is asking what it returns. How would you show whether the work is driving adoption, and how would you defend the budget when you cannot attribute cleanly?
Split your numbers into reach, engagement and behaviour, and only ever claim the behavioural ones as results. Where attribution is impossible, defend the programme with cohort comparisons, holdouts and counterfactual costs rather than an invented revenue figure.
When does ROC-AUC mislead you, and when does a precision-recall curve mislead you instead?
ROC-AUC vs precision-recall is a metric choice for imbalanced classification. ROC-AUC measures ranking across all negatives, while PR curves expose alert precision and recall at the positive-class prevalence.
The business says "we need a dashboard" and gives you a three-week deadline. Walk me through what you do first.
Find the decision the dashboard is meant to support and the event that triggered the request, define every metric word precisely, check the data exists before promising the picture, and take back a one-page note that fixes scope against the date instead of accepting the date against unknown scope.
Which metrics tell you whether retrieval is working?
Recall at k is the primary one, since a passage that never reaches the prompt cannot be used; precision at k measures wasted window; MRR or NDCG measure ordering. Read them beside the groundedness of the answer, because high recall with bad answers points at generation or context ordering.