Loading...
Loading...
Browse 11 real-world technical and behavioral interview questions about Observability. Review scenarios, edge cases, and architectural best practices.
Every distinct combination of label values is a separate time series with its own index entry, so an unbounded label multiplies series count by the number of users rather than adding one dimension. Metrics are the wrong tool for per-entity questions: keep labels bounded, move per-user debugging to exemplar-linked traces, and guard against the next unbounded label.
Ask the index directly whether the old text is still there, because that single check splits the problem into change detection, queue or write failure, a retire step that left duplicate chunks, or a downstream cache. Then fix the reason you learned about it from a user rather than an alert.
Split the measurement into time-to-first-token and inter-token latency, because they come from different phases. The first is driven by prompt length and queueing and is fixed with shorter prompts, prefix caching and streaming; the second is the decode loop, fixed only by generating fewer tokens.
Walk three gates in order for that one request — was the supporting passage retrieved, did it survive into the rendered prompt, does the answer contradict it. Each gate blames a different stage, which is why the diagnosis depends on having logged both the retrieved chunk ids and the prompt as sent.
Decide what may cross the provider boundary and redact before it does, checking the provider's retention and training-use terms rather than assuming them. Then treat the debugging trace as a second copy of user data, with its own retention, access control and deletion obligations.
Flat traffic and a tripled bill means cost per request tripled, so hunt what changed per request: a larger model version, more feature reads per prediction, retries multiplying calls, or warm capacity left running. Per-request cost attribution is what makes this answerable.
Metrics tell you something is wrong and when it started, traces tell you which hop owns the latency, logs tell you what happened to one request. Cardinality is what makes metrics expensive, sampling is what makes traces affordable, and the goal is answering new questions from data already held.
A flat p50 rules out a uniform slowdown, so the tail belongs either to a subset of requests or to a queueing effect. Partition the latency histogram by instance, endpoint and tenant before naming a cause, then confirm with traces, GC logs and pool-wait metrics.
An SLO defines the reliability users should experience, and the error budget is the allowed gap between that target and perfection. Use burn rate to decide when reliability work must outrank feature delivery.
An architectural blueprint for unified observability pipelines, focusing on edge processing, intelligent tail-based sampling, and vendor-agnostic telemetry.
Trace every request with prompt and model versions, token counts and cost, then add sampled groundedness grading, refusal and retry rates, validation failures and implicit user signals. Alert on countable rates, because an averaged quality score moves too little and too late to page anyone.