You are pushing a configuration change into the mobile core tonight. How do you stop one change taking the whole network down?

Redundancy does not protect you from your own change

Start by separating the two failure classes, because conflating them is what makes networks with impressive resilience diagrams fall over completely.

Hardware and site failures are random and independent, and the standard answers work: redundant elements, geographic separation, pooled capacity, automatic failover. A configuration error is neither random nor independent. It is applied deliberately, correctly, to every member of the redundant set, because that is what consistency means in an operational estate. Both halves of the pair receive it, both behave identically, and failover has nowhere to fail over to. The same is true of the software-defined core, where the appeal of automation is that one intent propagates everywhere quickly, which is also precisely the property that makes a bad intent propagate everywhere quickly.

So the containment mechanism has to live in the change process, and the design question is how much of the subscriber base a single push can affect before someone notices. That number, stated out loud, is the answer to the interviewer's question. If it is "all of them", no amount of node-level redundancy changes it.

Staged exposure, and what a stage is in a network

Staging a change in a mobile core is not as simple as a percentage of traffic, and knowing why is a differentiator.

The natural units are geographic and functional. You can take one region, one site, one pool member, or one instance of a control-plane function such as the access and mobility management function or the session management function, and apply the change there while its peers remain on the previous configuration. That immediately raises the awkward question of whether the change is compatible with the old configuration running alongside it, because for the duration of the rollout it must be. A change that is only correct once every element has it is a change that cannot be staged, and recognising that class early — usually something touching a shared data model, a protocol version or a policy that must be consistent across a pool — is what tells you the rollout plan needs a different shape, typically a compatibility release first and the switch afterwards.

Between stages sits soak time, and soak time is the control people delete under schedule pressure. Its purpose is not to wait; it is to let the slow failure modes appear. Many core faults are only visible with a full traffic mix and a full day's cycle: the roamer attaching at a boundary, the handover between an upgraded and a non-upgraded site, the busy-hour signalling load, the batch that runs at three in the morning, the memory that leaks over eight hours. A rollout that completes in one maintenance window has skipped every one of them, and the change looks successful because nothing has had a chance to fail yet.

Judge the stage by service, not by status

The next mistake is choosing the wrong evidence to proceed on. An element that is up, has no alarms and reports healthy is a very weak signal in a network, because the interesting failures degrade service while every device claims to be fine.

The gate on each stage should be subscriber-visible measurements from the region you have just changed, compared against an unchanged region as the control: attach and registration success, session establishment success and setup time, call setup success and drop rate, handover success across the boundary between changed and unchanged areas, throughput at the cell edge, and the volume of usage records arriving at mediation, which is a surprisingly good end-to-end detector because it depends on almost everything working. Add synthetic tests originating from real devices in the affected area rather than from a probe inside the data centre, because the parts of the path a probe skips are the parts customers use.

The comparison against a control region is the part that gets missed. Absolute numbers move for reasons unrelated to your change — a football match, a public holiday, weather — so a threshold on the changed region alone produces both false alarms and false confidence. The differential is the signal.

The rollback has to be reachable and it has to have been run

Two properties turn a rollback from a paragraph in a plan into a control that works.

It must be executable when the traffic path is broken. If the engineers reach network devices over the same network they have just flooded, or authenticate against a directory that is now unreachable, or need a VPN that rides the affected core, then the rollback exists only for failures mild enough not to need it. That means an out-of-band management path, credentials that are available offline, and at least one person who can physically reach a site. It also means an incident channel that does not depend on the mobile network the company operates, which sounds absurd until the day the engineers are all on their own network's SIMs.

And it must have been performed, not merely written. A rollback is a change like any other, with its own failure modes: configuration that cannot be reversed because it migrated state, a device that will not accept the previous version, a dependency that has moved on. Practising the reversal in a lower environment tells you which of those apply. The stronger form is time-bounded automatic reversion, where a change applies for a fixed period and reverts itself unless positively confirmed, which converts "someone must notice and act" into "someone must actively keep it".

Emergency calling is a different obligation

Availability targets are commercial; emergency calling is not. In most markets the ability to reach emergency services is a regulatory condition of operating, so a change that can affect it is not in the same category as one that can affect data throughput, however much larger the second population is.

Practically, that means knowing which elements in the path are required for an emergency call to connect and to be located, treating any change touching them as a higher class of change with its own approval, and testing that path explicitly on every stage rather than inferring it from general service health. It also means that a rollout plan should say what happens to emergency calling if the change fails, because the answer "the network is down so nothing works" is the answer a regulator will ask about first.

What the Rogers outage teaches about the change window

In July 2022 a routing configuration change on Rogers' network removed filtering that had been holding traffic back, the core was flooded, and nationwide mobile and fixed service failed for most of a day, including access to emergency calling. Because emergency service was affected it became a regulatory matter rather than a customer-service one, and much of the subsequent scrutiny concerned how a single change could reach the entire network at once and how difficult recovery became when the paths needed to fix it ran over the network that had failed.

Take three things from it into any answer. A single change that can reach the whole estate is an architectural fact, not a process failure, and no review board changes it. The management and recovery path must not share a fate with the traffic path. And where emergency calling and fixed-line service depend on the same core as mobile, the convergence that saved money has also concentrated the risk, which is a trade-off worth stating rather than discovering.

The question is never whether the change was reviewed. It is how many subscribers one push can reach before anyone measures service, and whether you can get back with the network in the state your change put it in.