Top 11 Observability Interview Questions
Browse 11 real-world technical and behavioral interview questions about Observability. Review scenarios, edge cases, and architectural best practices.
Engineers added a user_id label to a few metrics and the observability bill went up tenfold. Explain what actually happened, and how you would fix it without losing the ability to debug.
Every distinct combination of label values is a separate time series with its own index entry, so an unbounded label multiplies series count by the number of users rather than adding one dimension. Metrics are the wrong tool for per-entity questions: keep labels bounded, move per-user debugging to exemplar-linked traces, and guard against the next unbounded label.
A document was updated an hour ago and the assistant is still quoting the old version. Walk me through the diagnosis.
Ask the index directly whether the old text is still there, because that single check splits the problem into change detection, queue or write failure, a retire step that left duplicate chunks, or a downstream cache. Then fix the reason you learned about it from a user rather than an alert.
Users say the LLM feature in your product feels slow. How do you work out what to fix?
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
A user reports a wrong answer. How do you work out whether retrieval or generation failed?
Walk three gates in order for that one request — was the supporting passage retrieved, did it survive into the rendered prompt, does the answer contradict it. Each gate blames a different stage, which is why the diagnosis depends on having logged both the retrieved chunk ids and the prompt as sent.
How do you handle personal data in prompts, logs and traces for an LLM feature?
Decide what may cross the provider boundary and redact before it does, checking the provider's retention and training-use terms rather than assuming them. Then treat the debugging trace as a second copy of user data, with its own retention, access control and deletion obligations.
Your inference bill tripled last month and traffic was flat. How do you find out why?
Flat traffic and a tripled bill means cost per request tripled, so hunt what changed per request: a larger model version, more feature reads per prediction, retries multiplying calls, or warm capacity left running. Per-request cost attribution is what makes this answerable.
Latency on a checkout endpoint has tripled and nobody knows why. What should your metrics, logs and traces already have told you before anyone opens an editor?
Metrics tell you something is wrong and when it started, traces tell you which hop owns the latency, logs tell you what happened to one request. Cardinality is what makes metrics expensive, sampling is what makes traces affordable, and the goal is answering new questions from data already held.
Your p99 latency jumped tenfold after a deploy while p50 is unchanged. Diagnose it out loud.
A flat p50 rules out a uniform slowdown, so the tail belongs either to a subset of requests or to a queueing effect. Partition the latency histogram by instance, endpoint and tenant before naming a cause, then confirm with traces, GC logs and pool-wait metrics.
How would you set an SLO for a service, and what is an error budget for?
An SLO defines the reliability users should experience, and the error budget is the allowed gap between that target and perfection. Use burn rate to decide when reliability work must outrank feature delivery.
How do you design a petabyte-scale telemetry pipeline that decouples generation from storage while aggressively curbing backend SaaS costs?
An architectural blueprint for unified observability pipelines, focusing on edge processing, intelligent tail-based sampling, and vendor-agnostic telemetry.
What do you instrument and alert on for an LLM feature running in production?
Trace every request with prompt and model versions, token counts and cost, then add sampled groundedness grading, refusal and retry rates, validation failures and implicit user signals. Alert on countable rates, because an averaged quality score moves too little and too late to page anyone.