Engineers added a user_id label to a few metrics and the observability bill went up tenfold. Explain what actually happened, and how you would fix it without losing the ability to debug.

Short answer

A label is not a column you can filter later — every distinct combination of label values creates its own time series, stored and indexed independently. Adding user_id to a metric does not add one dimension; it multiplies the series count by the number of users. One metric with three labels becomes two million metrics, and you are billed per series. Metrics answer aggregate questions; per-entity questions belong to traces and logs.

The multiplication

Cardinality is the product of the distinct values of every label, not the sum.

http_requests_total{method, status, endpoint}
   5 methods × 8 statuses × 40 endpoints            =      1,600 series

http_requests_total{method, status, endpoint, user_id}
   5 × 8 × 40 × 500,000 users                       = 800,000,000 series

The request volume did not change. Not one extra request was served. The number of things being counted separately went up by a factor of half a million, and each of those things needs an index entry, a chunk of memory in the head block, and a row in whatever your vendor counts for billing.

This is why the bill moves so much more than the traffic. In a Prometheus-style system, the dominant cost of a series is not its samples — it is the inverted index mapping label values to series, which lives in memory and grows with distinct values. A series that receives one sample and is never written again still costs nearly as much as a busy one for the duration of its retention.

Why the cost persists after you remove the label

Worth knowing because it surprises people mid-incident: deleting the label stops new series being created, but the ones already ingested remain until retention expires. Churn makes this worse — if user_id appears in a metric emitted per session, each new session mints a series that goes permanently inactive minutes later, so you accumulate dead series continuously. Cardinality problems have a tail measured in weeks, which is an argument for prevention rather than cleanup.

What a label may contain

The rule that survives contact with reality: a label value must come from a small, bounded set that you control, and the bound must not grow with your business.

Safe: HTTP method, status code, region, environment, service name, a normalised route template, an enum of error categories. Each has a ceiling you can state.

Unsafe: user id, session id, request id, order id, email, raw URL path, full error message, SQL statement, customer name. Each grows without limit, and the last two grow in ways nobody predicts — a stack trace embedded in an error label creates a new series for every distinct trace.

The raw URL path deserves particular attention because it is the most common accident. /orders/8123/items and /orders/8124/items are two label values and eventually two hundred thousand. The fix is to label with the route template — /orders/{id}/items — which the router already knows. Any instrumentation that takes the path straight from the request object has this bug latent in it.

Where the per-user question should go

The engineers who added user_id had a real need: when a specific customer complains, someone must be able to see what happened to them. Metrics were simply the wrong instrument, and the answer is not to refuse the requirement but to route it.

Traces are built for this. A trace is per-request by construction, carries arbitrary high-cardinality attributes without multiplying anything, and is sampled so the cost is bounded by sampling rate rather than by user count. user_id as a span attribute is entirely appropriate; as a metric label it is not.

Logs serve the same purpose for events that are not spans, with the same property — high-cardinality fields are cheap because you are not maintaining a time series per value.

Exemplars are the bridge, and mentioning them is a strong signal. An exemplar attaches a sample trace id to a metric bucket, so a spike in the p99 latency histogram links directly to an actual slow trace. You keep the aggregate metric cheap and low-cardinality, and you can still jump from "the p99 got worse at 14:20" to a specific request that was slow, without any per-user series existing.

For the narrower ask — per-customer latency for the top twenty accounts — the workable compromise is a bounded label: an explicit allowlist mapping those twenty account ids to a tier or account label and everything else to other. Twenty-one values is bounded and stays bounded, and it is a deliberate decision rather than an accident.

Finding the damage before deleting anything

Do not start by guessing which metric is responsible. Rank by series count first:

topk(20, count by (__name__)({__name__=~".+"}))

and, for a suspect metric, find which label is doing it:

count(count by (user_id) (http_requests_total))

That number is the multiplier. Most vendors expose the same ranking natively, and TSDB status pages report the top label-value cardinality directly. Working from that list means you delete the two metrics causing the problem instead of a dozen that were fine.

Preventing the next one

Cleanup without a control means this recurs the next time someone debugs in production. Three mechanisms do most of the work.

Enforce a relabel or attribute-filter rule at the collection layer that drops known-dangerous labels regardless of what applications emit. This is the only defence that works against code you did not review.

Set a per-metric series limit so a runaway metric is capped rather than allowed to consume the cluster. Prometheus supports sample and label limits per scrape target; most vendors have an equivalent. A metric that hits the cap and drops data is a much better outcome than an ingestion tier falling over.

Add a budget alert on series growth rather than on the bill, so a tenfold increase is visible within an hour instead of arriving with the invoice a month later. The cost signal you want is leading, not trailing — which is the same argument as monitoring error budget burn rather than monthly availability.

---

© 2026 Preptima. Originally published at preptima.com.