Every dashboard is green and a customer tells you checkout has been broken for an hour. How does your monitoring tell nothing-is-wrong apart from nothing-is-reporting?

Green is two different states wearing one colour

Start by naming the ambiguity precisely, because it is the whole question. A chart showing no errors and a chart showing no data render almost identically, and a threshold alert defined as "fire when the error rate exceeds two per cent" cannot fire when the error rate is unknown. Every rule of that shape is silently conditional on the data arriving, and that condition is invisible in the rule.

So the design principle is that absence is a signal. Not a gap to be interpolated over, not a quiet period, but a distinct state that must be detectable and alertable. Until absence is a first-class condition in your alerting, your monitoring is only capable of telling you about failures that were kind enough to leave a trace.

The public example is uncomfortable and worth having ready. In August 2003, a race condition in a utility's alarm software stopped operators receiving the alerts they were relying on, and that loss of visibility contributed to a cascading failure across the north-eastern power grid. The monitoring system is itself a system, it fails, and it tends to fail quietly at exactly the moment its output matters most. A design that cannot distinguish nothing being wrong from nothing reporting is not finished.

Make something arrive every minute whether or not anything happens

The cheapest fix is a heartbeat, and it works because it inverts the logic. Emit a signal on a fixed schedule that has nothing to do with traffic, and define an alert that fires when that signal has not been seen for some interval. Now silence has a consequence.

There are two layers of this and both are worth having. Per component, a liveness signal — Prometheus's scrape produces an up series for each target for exactly this reason, and absent() lets you assert that a series you expect exists at all. And end to end, a deliberate always-firing alert routed to a service that pages you when it stops arriving, the dead man's switch. That external service is doing something your own stack structurally cannot: asserting that the entire path from metric emission through storage through rule evaluation through notification is intact, by observing its output rather than its parts.

The reason the outermost check must be external is worth stating plainly. Any monitor that runs on the infrastructure it monitors shares the failure mode that takes it out. An alert rule evaluated by a component that has crashed cannot report that it has crashed. The independence you need is not conceptual, it is physical: a different provider, a different network path, a different credential store, so that the mechanism which tells you the observability plane is broken has no ingredient in common with it.

The pipeline fails in shapes that produce plausible-looking numbers

Total silence is the easy case, because it is at least conspicuous once you have a heartbeat. The harder failures are partial, and they are dangerous because they yield charts that look ordinary.

If an agent's buffer fills under load and it drops samples, your data thins out precisely when the system is busiest. That biases everything computed from it: a p99 derived from the surviving samples is optimistic, because the slow requests are disproportionately the ones that occurred during the pressure that caused the dropping. You get a latency chart that improves during an incident. Instrument the collector's own drop counter and treat a non-zero value as a data-quality alert, not as a curiosity.

If a subset of instances stops reporting — a rollout where the new image lacks the agent, a node whose disk filled, a network policy change — your aggregate is now computed over the healthy remainder. A fleet where four of forty instances are failing every request can show a fine error rate if those four are also the ones not reporting. The defence is to assert cardinality: alert when the number of reporting instances differs from the number the orchestrator says should exist, which is a comparison between two independent sources rather than a threshold on one.

And if timestamps come from the emitting host rather than from ingestion, clock skew relocates data. Events land in the future and are invisible on a dashboard showing up to now, or land in the past behind the alert's evaluation window and never trigger anything.

Instrumentation regressions are the ones nobody catches

The failure with no operational cause at all is a code change. Somebody refactors a handler and the timer no longer wraps the slow call. Somebody renames a label from status_code to status, and every alert, dashboard and recording rule that grouped by the old name now matches nothing. Somebody adds a sampling decision to reduce cost and the trace you would have needed no longer exists.

None of these produce an error. The deploy is green, the tests pass, and the observability of that path is simply gone until the next incident discovers it. Three practices help, and naming any of them is a strong signal because most candidates never consider instrumentation as something that can regress. Treat the alert rule set as code with tests, asserting against recorded data that each rule's query returns a series at all. Alert on the disappearance of a series you depend on, which is the same absent() discipline applied to the metrics your alerts are built from rather than only to targets. And review instrumentation changes with the same seriousness as the logic they wrap, because deleting a metric is deleting the only thing that will tell you the deletion mattered.

The signal that does not depend on your system being right

Everything above still shares one assumption: that your code is the thing producing the evidence. Break that assumption with at least one measurement taken from outside.

A synthetic probe that performs the real user journey on a schedule from outside your network — resolve the name, complete the handshake, sign in, add to basket, reach the payment step — fails when the flow is broken regardless of what your internal metrics believe, and it fails when DNS, the certificate, the CDN or the load balancer is the problem, which is the class of fault your in-process instrumentation is structurally blind to because the request never reaches it. Its weakness is coverage: one path, one location, low frequency, and no visibility into the customer whose particular data triggers the bug.

Real-user monitoring is the complement. Reporting from the client tells you what actual browsers experienced, including the requests that never arrived, which is the only place a total ingress failure is visible at all. Its weakness is that it depends on the client being able to reach your collector, so it is not a substitute for the synthetic check either.

The answer to the question as posed is therefore procedural as well as architectural. When a customer reports something your dashboards deny, do not start by doubting the customer. Establish first whether the signal exists — is the series present, is the heartbeat current, is the instance count right, is the collector dropping — and only then interpret its value. Reasoning confidently from a chart that is green because it is empty is the most expensive mistake available during an incident.

Absence of data must be an alertable state, the thing that watches the watcher must not run on what it watches, and at least one of your signals should come from outside the system entirely — because a threshold alert on a metric that stopped arriving is not a quiet alert, it is a disabled one.