Latency on a checkout endpoint has tripled and nobody knows why. What should your metrics, logs and traces already have told you before anyone opens an editor?

Three signals, three different questions

The signals are not three views of the same data and they are not substitutes, and the fastest way to demonstrate that is to say which question each one answers.

Metrics answer whether something is wrong and since when. They are pre-aggregated numeric time series, cheap to store at any request volume because cost scales with the number of series rather than the number of requests, which makes them the only signal you can afford to keep at full fidelity for a year. They are what alerts fire on and what a dashboard shows. They cannot tell you which request was slow, because by the time you queried them the individual requests no longer exist.

Traces answer which hop owns the latency. A trace is a tree of spans sharing a trace id, propagated across process boundaries in a request header, each span carrying a start time, a duration and attributes. For the tripled checkout latency this is the signal that resolves it: the waterfall shows whether the time is in your handler, in a downstream service, in a database call, or in queueing before your code ran at all. Nothing else answers that question, and teams that skip tracing end up inferring it from timestamps across five services, which is tracing done badly by hand.

Logs answer what exactly happened to this one request. They are the only signal with room for arbitrary detail — the payload that failed validation, the branch taken, the exception. Their cost scales with request volume, which is why they are the signal that gets sampled, truncated and eventually deleted, and why unstructured logs are close to worthless at scale: a string you cannot filter or aggregate on is only findable if you already know what you are looking for.

The practical consequence is that these should not be three disconnected systems. Emit the trace id in every log line and attach exemplars to your metrics so a spike on a chart links directly to a trace that constitutes it. The navigation from "the graph is wrong" to "this request shows why" is the whole value, and it comes from correlation identifiers rather than from any one tool.

Cardinality is the bill

A metric is not one thing. http_requests_total with labels for method, status and route is one time series per distinct combination of label values, and each series is stored, indexed and kept in memory independently. Five methods, eight statuses and forty routes is 1,600 series, which is nothing. Add customer_id across four thousand tenants and it is 6.4 million, and you have not added information proportional to that cost — you have added a dimension nobody queries by.

The rule to state plainly is that anything unbounded must never be a metric label: user id, request id, session id, email address, full URL path with identifiers in it, error messages containing a stack frame. Each of those turns a metric into a log with worse ergonomics and a much higher price. The failure is also not gradual; a single deploy that adds one high-cardinality label can push a Prometheus server past its memory limit and take out the monitoring for everything else, which is the cruellest possible time to lose it.

What you do instead is keep dimensions in metrics bounded and deliberate, and move genuine high-cardinality questions into traces and structured events, where per-record storage was the model all along. If you need per-customer latency, that lives in a wide-event or trace store you can group by customer at query time, not in a series-per-customer metric.

Why you cannot average a p99

This is where otherwise strong candidates quietly say something false. Percentiles are not linear, so the p99 of a fleet is not the mean of each instance's p99 and there is no arithmetic that recovers it from those numbers. If forty instances each report a p99 of 200 ms, the fleet p99 could be 200 ms or considerably worse, and you cannot tell which.

The fix is structural: store a histogram, not a pre-computed quantile. Histogram buckets are counters, counters are additive across instances and across time, and the quantile is estimated from the summed buckets at query time. A client-side summary that computes its own quantiles is the thing that cannot be aggregated, and the difference between the two is one of the few genuinely load-bearing details in metric instrumentation.

The corollary matters for the checkout incident. Bucket boundaries are chosen when you instrument, and a histogram whose largest bucket is 1 second cannot distinguish a 3-second regression from a 30-second one. Choose boundaries against the SLO you care about, and expect to be wrong once.

Sampling traces without losing the interesting ones

Tracing every request at high volume is affordable to generate and expensive to store, so you sample, and where you make the decision determines what you can still see afterwards.

Head-based sampling decides at the root span, before anything has happened, and propagates the decision in the trace context so every service agrees. It is cheap, requires no coordination and keeps traces complete. Its defect is unavoidable: at the moment of the decision you do not know whether the request will fail or take nine seconds, so at a 1% rate you keep 1% of the errors too, and the rare pathological request that explains the incident is almost certainly discarded.

Tail-based sampling buffers spans in a collector until the trace looks complete, then decides — keep everything that errored, everything slower than a threshold, a small random sample of the healthy remainder. It gives you exactly the traces worth having. The costs are real and worth naming: every span of a trace must reach the same collector instance, so you need a routing layer that shards by trace id; the collector holds spans in memory for a decision window, which is a memory budget and a hard timeout; and a trace that straddles the window is decided on incomplete information.

The usual production answer is both. Head-sample at a modest rate to keep a representative baseline for latency comparison, and tail-sample aggressively for errors and outliers. Two constraints apply regardless of scheme. Sampling must be consistent across services or you get truncated traces that look like missing dependencies, which is worse than no trace. And you must record the sampling rate alongside the data, because a count derived from sampled traces is wrong by that factor and someone will eventually build a dashboard on it.

Designing so the next incident does not need a deploy

The failure this question is really probing is the one every team has lived: the incident starts, you look at what you have, the answer is not in it, and the fix is to add a log line and ship it — which means waiting for a build, a deploy, and for the problem to happen again. Everything above is in service of not being in that position.

Three habits get you there. Instrument every outbound dependency at the moment you write the call, not later, with request rate, error rate and duration each — the gap you will regret is always a dependency nobody thought was interesting. Emit wide structured events with many attributes rather than narrow pre-aggregated counters, because an event with thirty fields can be grouped by a dimension you had not thought of, while a counter can only answer the question it was created for. And make verbosity a runtime control, so raising log level for one tenant or enabling 100% tracing for one endpoint is a configuration change rather than a release.

Then the honest closing point: this is a budget, not an ideal. Observability spend that rivals production spend is a real and common outcome, so the deliberate posture is high-fidelity data with short retention plus aggressive downsampling for the long tail — keep raw events for days, histograms for months, and a handful of SLO series for years.