Observability at Scale interview questions

Metrics, logs, traces, cardinality control, and designing systems that can be debugged live.

4 questions in System Design.

System DesignA 24-minute read on how the topic fits together, before the questions.System design fundamentals60 short answers on one page, for revising rather than studying.

hard4

hardConceptScenarioDesign

Engineers added a user_id label to a few metrics and the observability bill went up tenfold. Explain what actually happened, and how you would fix it without losing the ability to debug.

Every distinct combination of label values is a separate time series with its own index entry, so an unbounded label multiplies series count by the number of users rather than adding one dimension. Metrics are the wrong tool for per-entity questions: keep labels bounded, move per-user debugging to exemplar-linked traces, and guard against the next unbounded label.

5 minmid, senior, staff

hardScenarioDesign

Every dashboard is green and a customer tells you checkout has been broken for an hour. How does your monitoring tell nothing-is-wrong apart from nothing-is-reporting?

A green dashboard and a dead pipeline look identical, so absence of data has to be an alerting condition in its own right: heartbeats that fire when they stop arriving, an alerting path that does not share infrastructure with production, and at least one signal produced outside the system being measured. It also connects monitoring pipeline to the point an interviewer is testing.

6 minmid, senior, staff

hardDesignScenario

Latency on a checkout endpoint has tripled and nobody knows why. What should your metrics, logs and traces already have told you before anyone opens an editor?

Metrics tell you something is wrong and when it started, traces tell you which hop owns the latency, logs tell you what happened to one request. Cardinality is what makes metrics expensive, sampling is what makes traces affordable, and the goal is answering new questions from data already held.

6 minsenior, staff, lead

hardScenarioDesign

How do you design a petabyte-scale telemetry pipeline that decouples generation from storage while aggressively curbing backend SaaS costs?

An architectural blueprint for unified observability pipelines, focusing on edge processing, intelligent tail-based sampling, and vendor-agnostic telemetry.

2 minsenior, staff