A planned upgrade window opens and correlation stops working, so operations sees thousands of uncorrelated alarms. What is wrong with your inventory and change data?

Correlation is a graph traversal, so it fails the way graphs fail

Reducing a wall of alarms to a root cause is not pattern matching on alarm text. It is a walk over the inventory graph: this alarm is on a port, the port is contained by a card, these other alarming elements sit downstream of that card over links inventory knows about, these resource-facing services consume those resources, and these customers hold the customer-facing services above them. Every step of that reasoning is a fact read out of inventory.

Which means correlation has a dependency that never announces itself. When the graph is wrong, no error is raised and no rule reports a miss — the engine simply finds no parent for each symptom and forwards all of them as independent faults. The visible failure is a flooded console and an assurance team concluding the correlation engine is useless, when what happened is that its input was stale.

A planned upgrade is the worst possible moment for that, because it is the one activity that mutates the exact subgraph being traversed, and it does so while alarm volume from that subgraph is at its peak.

What the upgrade does to the graph

Three things go wrong, in roughly this order of frequency.

Identity changes. A card is replaced and the new one presents its ports under identifiers the record does not hold, an element is renamed as part of a standardisation, a link is re-homed onto a different aggregation device. Alarms now arrive citing resources inventory has never heard of, and an unknown resource has no parent, no service and no customer, so it cannot be correlated or impact-assessed at all.

Timing is backwards. Inventory in most operators is maintained as an as-built record, updated after the work by the team that did it, sometimes days later. During the window the record describes the estate as it was, the network reports the estate as it is becoming, and correlation is reading the one that no longer exists.

Connectivity is modelled thinly. Containment — card in chassis, port on card — is usually accurate because it is physical and static. Connectivity between elements, which is what lets an alarm on a core router explain an alarm on a base station two hops away, is the part that decays, and it is the part the traversal needs most.

Change data belongs in inventory, not in a ticket description

The remedy is to stop treating planned work as a calendar entry and start treating it as structured data about resources. A change record should carry the identified set of resources it will touch, the planned target state of each, and the window. That set is the object everything else keys off.

Once you have it, three capabilities follow immediately that no amount of alarm-side cleverness provides. Suppression can be scoped to the resource set, so alarms from resources inside the change are attributed to it and alarms from anywhere else stay fully visible. Impact can be calculated before the work rather than discovered during it, because traversing upwards from the resource set names the services and customers at risk, which is what proactive notification needs. And maintenance state can be set on those resources and propagated up the same graph, so the services above them are marked as under planned work — which is what stops the availability clock and applies the exclusion the service-level agreement almost certainly already contains.

Modelling a planned state alongside the as-built state is what makes this work across the window rather than only up to it. If the change record says port 3/0/1 will be replaced by a card whose ports are named differently, the correlation engine can resolve the new identifiers as soon as they appear instead of treating them as strangers.

Suppressing by time is the mistake that looks like the fix

Under pressure the obvious remedy is to mute the console for the duration of the window. It works, in the sense that the storm goes away, and it is wrong for a reason worth stating plainly: faults do not observe your calendar. A window that suppresses everything for two hours will suppress the unrelated power failure at the other end of the country that began forty minutes in, and nobody will look at it until the window closes.

Scope by the resource set instead, and accept that the scoping will be imperfect — the change will touch something nobody listed, and the leaked alarms are the signal that your resource set was incomplete. Keep suppressed alarms stored and searchable rather than discarded, because they are the only record of what the estate did during the change, and because a suppressed alarm that never cleared is exactly what you need to find afterwards.

The redundancy case deserves its own thought. When one of two protected paths is deliberately in the window, the service is not under planned maintenance, it is unprotected. Marking it as maintenance hides the fact that a single further fault takes the customer down, which is the risk the window actually created.

Silence is not confirmation

The last failure is the one that produces an incident the following morning. The window closes, the alarms have stopped, everyone goes home, and at eight o'clock a region has no service.

An alarm that is no longer being raised and an element that is no longer reporting are indistinguishable from the console. So closing a window needs a positive check rather than an absence: run discovery against the change's resource set, compare what came back against the planned target state recorded in the change, and confirm the services traversed from that set are passing a real test. Any resource in the set that discovery cannot see is an open finding, not a completed change.

That comparison is only possible because the change carried its intended target state as data. It is the same argument as everywhere else in inventory — the record holds intent, the network holds observed state, and the value is in comparing them deliberately.

Assurance quality is bounded by inventory quality, and the bound is invisible until the night it matters, because bad topology does not produce errors, it produces uncorrelated alarms.