Telecom OSS/BSS Interviews: A Masterclass for Architects and Engineers
What OSS and BSS interviews at operators, vendors and integrators are really testing: the order-to-activate spine, the product catalogue, mediation and charging, inventory, revenue assurance and portability, the TM Forum vocabulary, and the signals that separate someone who has worked in an estate from someone who has read about one.
What it is
An OSS/BSS interview is an interview about an estate rather than about a system. Nobody at an operator owns a codebase you could clone and read in a weekend. What exists is fifteen to fifty independently supplied products, some older than the people maintaining them, wired together by interfaces that were each negotiated separately, sitting over a physical network that changes on a hardware cycle and a commercial model that changes weekly. When an interviewer asks a question in this domain, they are asking whether you can reason about that shape, and almost everything separating a strong candidate from an adequate one follows from taking the estate seriously as the unit of design.
Business support systems are the commercial half: the party you are dealing with and the account that owes you money, the catalogue of what is on sale and what it costs, the order the customer placed, the charging that prices usage, the billing that turns a month of activity into a legal document, the collections that chase the ones who do not pay, and the settlement that pays partners their share. Operations support systems are the delivery half: the inventory of what exists in the network, the fulfilment and activation machinery that makes it carry a specific customer's traffic, the assurance stack that notices when it stops, and the workforce management that sends a van to a street.
The hand-off between the two is a service order. BSS has established what was sold, to whom, at what price, from what date; OSS decides which resources will realise it, makes the network do so, and reports back. That single interface carries an enormous amount of weight, and a large fraction of the interesting questions here are about what happens when it is crossed badly: an order that half-completed, a service live on the network with nothing billing it, a bill produced from a tariff that had already been superseded.
The order-to-activate spine
Every operator has a path from "a customer wants something" to "the network is carrying their traffic and the meter is running", and that path is the backbone of the domain. Learn it as a sequence of responsibilities rather than as a list of boxes, because the box names differ per vendor while the responsibilities do not.
A customer journey establishes intent and eligibility. Serviceability is checked, meaning some system answers whether this can be delivered at this address or to this subscriber at all. A quote is priced against the catalogue, an order is captured as a commercial statement and validated, and order management decomposes it into service-level and then resource-level work. Inventory reserves the resources that work will consume, and activation issues the commands that configure network elements and subscriber data stores. Charging and billing are told a subscription now exists, with a start date. Usage flows into mediation, is rated, and accumulates until a bill cycle closes and produces an invoice. Money comes in, or does not, and collections engages. All the way along, assurance is watching the service, and inventory is supposed to know what the truth is.
flowchart TD
accDescr: The order-to-activate path from capture in BSS through decomposition into service orders and inventory and feasibility, into provisioning and activation in OSS, then mediation rating and billing alongside assurance watching the service, both feeding reconciliation across the estate, with the fallout branch where feasibility says the address is not serviceable and the order fails at survey, and a dashed correction running from reconciliation back to inventory.
A[Order capture in BSS] --> B[Decomposition into service orders]
B --> C[Inventory and feasibility]
C -->|not serviceable| E[Fallout at survey]
C --> D[Provisioning and activation in OSS]
D --> F[Mediation rating and billing]
D --> G[Assurance watches the service]
F --> H[Reconciliation across the estate]
G --> H
H -.-> CTwo things in it are worth holding on to. The branch out of feasibility is where the estate lies to itself, because the order was quoted and sold against an answer that inventory gave and could not honour, and the customer learns about it on the day an engineer arrives. And the dotted return is not a reporting afterthought hung off the end: it is the mechanism by which the estate discovers that inventory, the network and billing disagree, and an answer without it has described a happy path rather than a system.
The domains and what each one owns
The product catalogue determines whether launching something is a configuration change or a project. It separates a product specification, describing what the customer conceptually buys and the characteristics it has, from a product offering, which wraps that commercially with prices, eligibility, geography and contract terms, and below those sit service and resource specifications describing what must exist for the product to function. The property that matters is not expressiveness but that ordering, fulfilment and rating all read the same model at runtime rather than each having it compiled in, which is the substance of where OSS ends and BSS begins.
Order management owns the commercial order as durable state and its decomposition into technical tasks. It is where sagas, compensation and idempotency live in this domain, and where the difference between an engineer who has built distributed workflows and one who has read about them becomes visible in about ninety seconds, as what happens between order capture and service live sets out. Provisioning and activation then own the act of making a network element serve a specific subscriber, and the recurring theme there is that network state becomes externally visible the instant it changes and cannot be enlisted in your transaction.
Network inventory owns the record of what exists, layered as physical equipment, logical resources riding on it and services consuming those, with assignments linking the layers. It is the least glamorous domain and the one whose defects cause the most downstream chaos, because everything else treats it as an oracle it can trust, and assurance quality is bounded by it since correlation is a traversal of the inventory graph.
Mediation owns the transformation of raw network usage into records the commercial systems can price: collection, decoding of element-specific and frequently binary formats, normalisation, deduplication, correlation of several records describing one event, enrichment, filtering and distribution. It is a data pipeline with hard correctness requirements and no ability to ask for the data again. Rating and charging then own the price of an event, offline for postpaid and online against a live balance for prepaid, and billing owns the periodic close, ending in an invoice that is a legal document and is never edited after issue.
Revenue assurance owns the question of whether the money that should have arrived did, and its instruments are reconciliations across the boundaries the estate is made of. A reconciliation you cannot run tells you two systems do not share an identifier, and an identifier they do not share is a design defect. Number portability, finally, owns the inter-operator process of moving a telephone number without changing it: externally timed, regulated, and coordinated through a mechanism your systems participate in but do not control, which is why interviewers like it, and the details are in when a customer ports their number away.
The standards vocabulary, and how much to carry
TM Forum publishes the shared dictionary this industry argues in, and interviewers use it whether or not they think of themselves as standards people. Three pieces matter.
The Business Process Framework, still universally called eTOM, is a hierarchical decomposition of the processes an operator runs. Its practical value in an interview is the vertical groupings, because those phrases have become the names of programmes: concept-to-market, lead-to-cash, order-to-activate, trouble-to-resolve, usage-to-cash. Placing a question inside one of those and saying what the adjacent processes need from you signals that you know where your work sits.
The Information Framework, called SID, is a shared information model. The parts to know cold are the specification-and-instance distinction and the product-service-resource layering: a product is what the customer bought, a service is the technical capability delivered, a resource is what the network supplies. Customer-facing and resource-facing services are distinguished for a real reason. The thing the customer thinks they have is one object, the several technical arrangements realising it are others, and the mapping between them is where the catalogue earns its money.
The Open APIs are REST interfaces for the boundaries between these domains, and they are why most greenfield BSS integration now looks broadly similar between operators. Product catalogue management, product ordering, service ordering, activation, service and resource inventory, service qualification, usage management, customer bill management, party and account management, and trouble ticketing each have their own specification. Do not memorise the numbers; learn the shape. The Open Digital Architecture is the surrounding target architecture positioning those APIs, and knowing it exists is enough unless the role is a standards role.
The framing to carry into the room is that the standards give you a vocabulary and a set of defaults, not a design. An interviewer who hears you recite the frameworks without a position on where you would deviate has learnt that you read the website.
Why we need it
This domain has its own interview culture because the constraints an operator works under are unlike those of a product company, and every one of them changes what a good answer looks like.
The estate is multi-vendor by necessity and cannot be rewritten. A charging system, a billing system, a catalogue, an order manager and an inventory are each large licensed products with their own data models, release calendars and support arrangements. You do not get to choose one consistent model; you get to choose the contracts between them and decide which system owns which fact. That is why so much of the seniority signal here is about boundaries and ownership rather than internal design: the internal design of the vendor products is not yours, and the interviewer knows it.
Money is the output, and it is audited. A bug in rating costs revenue in one direction or produces overcharging in the other, and overcharging at scale is a regulatory event and a restatement. Hence the domain's preoccupation with determinism, versioning, replay and reconciliation: a rated record must be explainable six weeks later from stored, versioned facts, and a candidate who reaches for audit trails unprompted is telling the interviewer something true about where they have worked.
Nothing can be paused. The bill run is a legal obligation with a date, number ports have industry-agreed timers, emergency calls must work. That removes an entire class of otherwise reasonable engineering answers, and it is why migration questions here are always phrased as "without stopping the business" and always want a strangler with cohorts rather than a cutover.
Real-world state escapes your transaction. A port assigned on aggregation equipment is a port no other order can take. A personalised SIM cannot be un-personalised. An engineer who visited has already cost money. A number ported away cannot be ported back by your workflow. Correctness here is convergence to intent plus an inspectable answer for every intermediate state, not atomicity, and the interviewer is listening for whether you know the difference.
Regulation shapes the design and not merely the paperwork. Notice periods before a price rise determine when a catalogue change can take effect, portability rules constrain which rejections are legitimate, data retention imposes requirements on which systems hold what, and consumer protection limits retrospective charging, which is why the unbilled-period question below is a commercial decision rather than an engineering one. Latency and volume are not uniform either: sub-second online charging decisions sit in the same estate as a monthly bill run and a roaming file that arrives three days late, so any design assuming a single consistency model is wrong about the parts that hurt.
Finally, the estate outlives generations of technology. A tariff sold on a decommissioned platform still has subscribers on it. The systems have been migrated three times and the migrations were each abandoned partway, which is why almost every real operator has two of something and a mapping table between them. Interviewers are hiring for the ability to work in that reality rather than to be affronted by it, and candidates who signal they will only work greenfield are screened out early and quietly.
What interviewers ask
Interviews here cluster into a small number of rounds, and each is scored on different observable behaviour. Knowing which round you are in tells you which behaviour to display.
| Round | The question usually looks like | The signal being graded |
|---|---|---|
| Vocabulary screen | Define OSS, BSS, mediation, convergent charging | Whether each definition carries a consequence, and whether the small distinctions are used correctly |
| Architecture | Draw the estate, place the boundaries | Whether every fact has one writer, and whether boundaries follow rate of change |
| Scenario and fallout | A production symptom, no cause given | Whether the symptom is bounded first, and which failures you notice are silent |
| Data and reconciliation | Find everyone in a bad state | Whether you can name the joining identifier, and whether findings get an owner |
| Migration | Modernise this without stopping the business | Cohorts and a facade, and whether you inventory the readers |
| Testing and release | Nothing can be tested end to end | Contracts at boundaries, test data as the hard half, synthetic orders in production |
| Stakeholder | Marketing want it in three weeks | Whether you hold the position and still hand over something actionable |
The vocabulary screen
Someone establishes early whether you know the domain at all, with questions that look like definitions: the difference between OSS and BSS, what mediation does, what convergent charging means. The scored signal is not the definition but whether it comes with a consequence attached. "BSS is customer-facing, OSS is network-facing" is a sentence from a slide. "The boundary survives because the two sides change at incompatible rates, so coupling them means a tariff change waits for a network release" demonstrates that you hold the concept as a design force. The interviewer grades this by asking "why does that matter" and seeing whether the answer was already implied.
The second signal is whether you use the industry's words correctly and unselfconsciously. Not acronym count, which is easy to fake, but small correctnesses: an invoice and a bill cycle are different things, a subscription and an account are different things, a service order is not a customer order, a resource can be reserved without being assigned. Misusing those reveals that your exposure was second-hand.
The architecture round
You are given an estate or asked to draw one, and several signals are graded at once.
Whether you assign each fact a single writer. The interviewer probes with "who owns the subscriber's address" or "which system decides the service is active", watching for whether you answer with a system or with a shrug. Two systems both authoritative for one fact is the defect that produces every reconciliation you will later be asked to build.
Whether you justify boundaries by rate of change and ownership rather than functional tidiness. Grouping by noun is a first-week answer; grouping by who changes it, how often, and under whose authority is what an architect does.
Whether your integration story has failure semantics. Naming an API is not designing an interface. They are listening for retry and idempotency semantics, timeout behaviour, whether the interface is synchronous because it needs to be or because it was easier, ordering guarantees on events, and what the consumer does with a response it never received. Describing only request and response shapes marks you as an integrator of happy paths. The corresponding senior signal is saying what you would not build, with a reason.
The scenario and fallout round
You are handed a production symptom and asked to reason: a customer barred despite paying, a bill run that died two thirds through, usage arriving three days late, a service live on the network that nothing bills. These are the highest-signal questions in the domain, and four behaviours are graded.
Whether you bound the symptom before proposing a fix, asking which systems could produce this appearance and ruling them in or out rather than reaching for the most familiar cause.
Whether you separate restoring service from fixing the cause from fixing the class. Three timescales, three owners, and conflating them is the commonest structural weakness in scenario answers. The barred-after-paying scenario exists to expose it.
Whether you notice which failures are silent. The failures that cost most in this estate have no complainant. A customer billed for a service that does not work rings up within a day; a customer with a working service and no subscription never rings up at all, which is why that class only exists if you go looking, as the network-active-but-unbilled case sets out. Candidates who reason about detectability rather than causes are marked up hard.
Whether you know where the decision stops being yours. The unbilled months, the goodwill credit, the retrospective charge are commercial and regulatory decisions with a technical input. Offering the input precisely and then escalating is correct; an engineer who quietly backdates a billing start date has made a pricing decision on the company's behalf.
The data and reconciliation round
Increasingly its own round, and where people who have genuinely worked in an estate separate visibly. Asked how you would find every subscriber in a bad state, prove a migration correct, or detect leakage, the signals are whether you can name the identifier joining two systems and admit when there is not one; whether your reconciliation runs on a cadence with an owner and a queue rather than producing a report somebody reads; whether you reconcile in both directions, because the two differences have different causes and different costs; and whether you treat a difference of one as a finding. One subscriber repriced without notice is still a compliance incident, and a tolerance threshold on a compliance reconciliation reveals that you have been graded on dashboards rather than outcomes.
The migration round
Almost guaranteed for senior and above, because almost every operator is mid-programme. Scored on whether you refuse a cutover and propose a strangler; whether you pick the first seam by data ownership rather than by which module is most unpleasant; whether you shadow-run and compare outputs before switching anything; whether you migrate by cohort and keep each subscriber's bill cycle intact; and, most tellingly, whether you inventory the readers. The reports, extracts and finance spreadsheets selecting directly from the monolith's tables are what prevents decommissioning, and naming them unprompted says you have been through this.
The testing and release round
Specific to this domain and badly served by generic preparation. The setup is that no environment holds the whole stack, so nothing can be tested end to end, and the question is how you get assurance anyway. Graded on whether you move assurance to the boundaries with contracts each side can test alone; whether you treat test data as the harder half rather than test infrastructure; whether you simulate the network deliberately and state what the simulator does not reproduce; and whether you propose continuous synthetic transactions in production as ongoing assurance rather than as a smoke test. The full treatment is in getting assurance when no environment holds the whole stack.
The stakeholder round
Non-technical in form, entirely technical in consequence. Marketing want a launch date, a vendor will not fix a defect, finance want the leakage number this week. What is scored is whether you can hold a technical position under commercial pressure without becoming an obstacle, and whether you translate. "The catalogue cannot express that" is a blocked conversation; "the catalogue can express it, provisioning needs an adapter which is six weeks, so here is what we can launch in two that recombines what exists" is a negotiation. Architect roles at operators weight this heavily.
How the same question is scored at different levels
Candidates frequently answer one level below or above the role. At mid level a correct mechanism is the bar: what mediation does, why activation needs idempotent tasks, how proration arises. At senior level mechanism is assumed and grading moves to trade-offs and failure modes, so you are expected to volunteer what breaks. At staff and above it moves again, to whether you can bound a problem across the estate and the organisation: which programme should not happen, which constraint is contractual rather than technical, what you would stop doing to fund what you propose. Candidates who bring only deeper mechanism to a staff loop are commonly downlevelled, and the feedback reads as "strong engineer, did not demonstrate scope".
Tells that you have not worked in an estate
Interviewers share a short list. Describing activation as a database write. Assuming one system can be authoritative for everything. Proposing a full-stack test environment as though the request had not already been refused. Treating prepaid as postpaid with lower latency. Talking about the catalogue as configuration without asking whether fulfilment reads it. Assuming a number can be un-ported. Believing an invoice can be edited. Offering a microservice decomposition without saying who writes each piece of data. None is fatal on its own if you recover, and all are noted.
Questions
These are phrased the way interviewers phrase them. The answers are the substance a strong response contains, not scripts to recite.
Boundaries, architecture and standards
Where does OSS end and BSS begin, and why is that line still there?
BSS owns the commercial reality: party and account, catalogue and price, order, charging, billing, collections, settlement. OSS owns the network reality: inventory, fulfilment and activation, assurance, and the field workforce. The hand-off is the service order.
The credit is in the justification. The line has survived every technology generation that was supposed to erase it because the two sides change at incompatible rates and under different authority. Commercial systems change when marketing wants a bundle, which is weekly, and the change is data. Network systems change when the estate or the standards change, which is a multi-year cycle, and the change is capability. Couple them and a tariff change waits for a network release while a hardware refresh threatens the bill run. Add that this is also a skills and supplier boundary, and you have a durable answer rather than two lists.
Draw me the systems in a mobile postpaid order and tell me who owns what.
Work down the spine and assign a single writer to each fact. CRM or a party management capability owns the customer and the account. The catalogue owns what is on sale and its price, and is read by ordering, fulfilment and rating rather than copied into them. Order management owns the commercial order as durable state, service ordering and activation own the technical work, and inventory owns numbers, SIM stock and network resources with their lifecycle states. The subscriber data store the core consults owns the network's live view. Charging owns the balance and the price of usage, and billing owns the invoice and the receivable.
Then name the facts commonly owned twice, because that is the real content, and for each say which copy is authoritative and how the others are kept honest.
| Fact | Authoritative owner | Where it also lives |
|---|---|---|
| Tariff and price | Product catalogue | Rating engine's own tariff store, channel caches, warehouse dimensions |
| Customer address | The canonical address authority | CRM, the order, field despatch |
| Service status | Order for intent, network for observed state | Inventory, the billing subscription |
| Balance and allowance | Charging | Self-service app, notification triggers |
What are eTOM, SID and the Open APIs, and how much of them would you use?
eTOM is a process decomposition whose vertical groupings have become the industry's programme names. SID is a shared information model whose most reused idea is the product-service-resource layering and the specification-versus-instance distinction. The Open APIs are REST interfaces for the boundaries between domains.
How much to use is the real question, and the answer is a position rather than an endorsement. They buy you a vocabulary, a canonical set of resource models and a large reduction in argument; they cost you when a vendor implements a subset and calls it compliance, when your data model does not fit the canonical one, and when a standard's generality forces you to model in three objects something your business has one of. Adopt the models and API shapes at the boundaries you integrate across, where the reduction in argument is worth most, and deviate internally with the deviation documented and its reason recorded. A candidate who says they would implement the standards exactly as published has not integrated a vendor product; a candidate who dismisses them has not had to negotiate an interface with one.
Why are product ordering and service ordering separate APIs?
Because they are separate lifecycles with separate consumers. A product order is a commercial statement addressed to a customer: this offering, this price, this date, this contract. It is customer-visible, needs coarse stable states a shop assistant can read out, and changes when the commercial model changes. A service order is a technical instruction addressed to the delivery estate: realise these characteristics on these resources. It is invisible to the customer, its states are as detailed as operations require, and it changes when the network changes.
Collapsing them has two consequences. Every change to the network estate becomes a change to a customer-visible contract, which is the coupling the OSS and BSS split exists to prevent. And one commercial order routinely produces several service orders with different durations, so a single state model must either lie to the customer or expose detail they cannot use.
How would you decompose a monolithic BSS without stopping the business?
A strangler with three commitments and one discipline. Pick the first seam by data ownership: a domain whose data can be assigned one writer, which in practice means the catalogue and then product ordering, both read-heavy for everything else and neither holding the ledger. Billing moves last, because it is stateful per cycle and every downstream financial process is calibrated to its output. Put a facade in front of the monolith and make everything new speak to that, because skipping it is how you finish with a distributed monolith whose only improvement is more network hops. Then move traffic by cohort: shadow-run and treat any output difference as a defect to be explained, cut over new customers who have no history and no open cycle, then migrate the base one bill cycle group at a time so no subscriber's cycle is split across two systems.
The discipline is that the strangler finishes only when the last reader of the legacy data has moved, and the readers you forgot are the reports and finance extracts. Inventory those at the start and give each a replacement.
Order management and order-to-activate
Walk me through what happens between an order being captured and the service being live.
Decomposition first: the commercial order is translated, using the catalogue, into concrete actions against named systems. For fixed broadband that typically means reserving an access port on the aggregation equipment serving the address, allocating subscriber credentials and writing them to the authentication store, pushing a policy profile so the line delivers the speed sold, creating the subscription in charging and billing, dispatching customer premises equipment, and possibly booking an engineer.
Then say why this is not a transaction. The systems share no transaction manager, and one participant is a van. Every side effect is externally visible the moment it lands. The elapsed time is wrong by orders of magnitude for a transaction, because an order waiting nine days for an appointment must survive orchestrator restarts and deployments, which means it is durable state advanced by events rather than a call stack.
So correctness comes from stable task identity derived from the order and the step, tasks that converge rather than command when the downstream system offers no idempotency of its own, declared compensating actions per task, a fallout queue with enough context for a human, and jeopardy monitoring that alerts before the promised date rather than after it.
A task in the middle of an activation fails. What happens to the ones that already succeeded?
Each task declares a compensating action, and what separates a strong candidate is saying that compensation is asymmetric and frequently lossy. Releasing a port and deleting an authentication entry are clean inverses. A personalised SIM cannot be un-personalised, an engineer who visited has already cost money, a ported number cannot be un-ported by your workflow. For those steps the compensating action is a business rule rather than an inverse function: raise a returns case, write off the visit, quarantine the resource until a human decides.
That is also why the customer-facing cancellation and the technical rollback are separate decisions. Whether to unwind is commercial: if four of five tasks succeeded and the fifth is an appointment the customer wants rescheduled, unwinding is the wrong answer. The workflow needs to be able to hold, not only to complete or compensate.
How do you make an activation safe against a duplicate submission?
Derive the key from the order and the step so it is stable across retries, redeploys and duplicate events, and require the downstream system to return the original outcome when it sees a key it has already processed. Where it will not, and many network-facing systems will not, the task becomes read-then-write and treats "already in the desired state" as success rather than as a conflict.
Say the distinction out loud, because interviewers listen for it: retryable and idempotent are different properties. A task is retryable if calling it again is safe; it is idempotent if calling it again leaves the same state. Only the second gets you through a partition in which you never learnt whether the first call landed, and that partition is the normal case at this boundary.
A field engineer fixes a stuck task by hand. How does your workflow find out?
It has to be told, and designing for that is the answer rather than forbidding it. Out-of-band action is legitimate at three in the morning, so systems of record should be reachable only through the fulfilment path wherever possible, and where they are not, the action must produce an event the workflow consumes so the record catches up rather than diverging permanently.
Skipping this produces a recognisable incident. An engineer who fixes the port and closes the ticket without telling the orchestrator has created a subscriber whose service works and whose record says it does not. Months later a cease against that order reclaims resources from a live customer, or a reconciliation reports the divergence and nobody can tell whether it was an original defect or somebody's repair, which is the worse outcome because it destroys the evidence you would use to find the generator.
Product catalogue and pricing
What does it take for launching a product to be a configuration change rather than a project?
The catalogue has to be executed rather than consulted, which means four consumers read it at runtime. Ordering reads the specification to build and validate the order, so a new characteristic appears in the journey without a front-end release. Fulfilment reads the mapping to decompose the order into service and resource actions. Rating reads prices and rules from the same offering the customer bought. Eligibility and validation are evaluated from the characteristic definitions rather than reimplemented per product.
Then name how this fails, because it is the interesting part. A team builds a genuinely configurable catalogue, models a new offering in an afternoon, and discovers that provisioning still has a hand-written branch per product type. The offering configures beautifully and cannot be delivered, so the launch still needs a fulfilment release, an activation release and a test cycle, which is exactly the cost the catalogue was bought to remove. A catalogue pays back only across the whole chain, and the discipline is honesty about which case you are in: recombining existing service and resource specifications needs no fulfilment code, and genuinely new network capability needs an adapter and always will.
Marketing want to raise the price of a plan a million subscribers are already on. What has to happen?
Separate three requests hiding in one sentence. Pricing new sales differently is a catalogue operation and nothing else. Repricing the existing base is a commercial act performed once per subscription, constrained by contract and regulatory notice obligations, landing on an invoice as a mid-cycle change with proration. Changing what the plan includes is both, and it is the sharpest, because an allowance is consumed continuously and a subscriber halfway through a period whose included volume shrinks has already used some of an allowance that no longer exists at that size. That has to take effect at a period boundary.
The versioning rule everyone recites, that a specification referenced by a live subscription is immutable, is not where these programmes fail. They fail on the extent of the version. The boundary must enclose everything determining what the customer pays: the price plan, the bundles and allowances granted, the discount and promotion rules referencing the offering, and the eligibility rules if they change what combination a subscriber holds. If a new version reuses a shared allowance object and someone edits its volume, both versions move and every existing subscriber is repriced by an edit nobody was watching. The full scenario goes further into the downstream copies of the tariff.
How would you prove a catalogue change is safe before releasing it?
Price the whole base twice and compare. Run pricing for every subscription against the changed catalogue in a shadow environment for a defined period, and compare subscription by subscription against the current catalogue's output. Every subscription whose price changes should be in the intended cohort, and every subscription in the intended cohort should change by the intended amount.
Anything else is a defect found before release rather than after: the shared allowance object that moved both versions, the discount rule that re-resolved, the promotion whose end date interacts with the new price, the handful of subscribers on a withdrawn offering from a decommissioned platform nobody remembered maps onto this specification. That comparison is only possible if pricing is a deterministic function of versioned inputs, which is the same property that lets you restart a failed bill run and defend a disputed line.
Provisioning, subscriber lifecycle and portability
A subscriber's service works on the network but nothing bills them. How did that happen?
Enumerate the shapes, because naming them is most of the diagnosis. An activation converged on the network and then failed further along its own task list, so the subscription creation in charging timed out or was never reached. An acknowledgement was lost in the other direction: the subscription was created, the response never arrived, the workflow retried, exhausted its budget and compensated away the only correct thing in the picture. A human configured the element directly to resolve a fault, or a bulk load populated the network from a spreadsheet, and nothing wrote back. Or a cease order half-executed, closing the subscription and leaving the network configuration in place.
Then explain why nobody noticed, which is the graded part. Provisioned-and-unbilled is silent in every direction and the only party who could report it has no reason to; the mirror-image failure announces itself within a day. So the quiet one exists only if a scheduled reconciliation compares subscribers configured on the network against those with an open subscription, in both directions, treating every non-empty difference as a finding with an owner.
So fix it. What do you write, and where?
Nothing directly into billing. Repair starts from the order, the only artefact recording what was supposed to happen. A stalled order is resumed: tasks are idempotent, replaying the ones that succeeded is a no-op, and the missing one runs. A completed order gets a corrective task re-driving the missing step with the same identity the original used, so no duplicate appears if the first attempt did land.
Writing the subscription straight into billing gets the customer billed, which feels like the goal, and leaves the order record still saying the activation failed, inventory still unaware of the assignment, and no trace that a defect occurred. And the case where no order exists must stay manual, because a job creating subscriptions from observed network configuration will convert a mistaken bulk load, a lab configuration on a production element, or a deliberately provisioned free service into a billed contract with a real customer's name on it. Billing someone for something they never ordered is worse than the leakage you were closing.
The customer has had free service for five months. What do you charge?
State the technical input precisely and then escalate, because that division is what is scored. You can establish when the service began carrying traffic, from usage records and the element's configuration history, so the size of the gap is a fact rather than an estimate. That is your contribution.
The decision is commercial and regulatory. Backdating the billing start date produces a first invoice several times the expected amount for a customer who did nothing wrong, which in many jurisdictions runs into limits on retrospective charging and everywhere runs into a dispute. Charging forward and writing off the gap is clean and costs real money. Most operators land on a policy: bill forward from a stated date, cap arrears at a defined number of cycles, notify before the invoice rather than with it. The point is that it is a policy applied consistently rather than a judgement made per ticket.
A customer ports their number away. What has to happen on your side?
Establish first that the port date is not yours. This is an inter-operator process governed by the market's regulator and coordinated through an agreed mechanism, with the gaining operator initiating, the losing operator validating, and a cutover time agreed between them.
sequenceDiagram
accDescr: A number-port sequence between customer, gaining operator, porting hub and losing operator, the port request carrying validation details to the hub, the hub asking the losing operator to release the number, an accept or a reject with a reason coming back, routing updated at the agreed time, and the losing operator ceasing on confirmation and closing billing.
participant C as Customer
participant G as Gaining operator
participant H as Porting hub
participant L as Losing operator
C->>G: Asks to port the number
G->>H: Port request with validation details
H->>L: Request to release the number
L-->>H: Accept or reject with a reason
H->>H: Routing updated at the agreed time
L->>L: Cease on confirmation and close billingOnly the last step is inside your own estate, and it is the only one not driven by an external timer, which is exactly why it is the step that goes wrong.
The failure modes are asymmetric, and saying so is the signal. Cease before the port completes and the customer has no service at all, because their number is not yet routed to the new operator and yours has stopped serving it, which may take them out of contact including for emergency calls. Cease late and the customer is fine because routing has moved, while you keep billing a subscription for a service you no longer provide, keep resources allocated, and cannot bill any usage generated meanwhile. Given that asymmetry, cease on positive confirmation of the port rather than on the scheduled time arriving, so late is the direction it fails in.
Then the details: validation scope is narrow and regulators generally allow rejection only for defined reasons, so a wrong or slow rejection is a compliance problem rather than a delay. Recurring charges prorate to the port date, advance-billed amounts become a credit, and the subscription must remain rateable after it stops being serviceable because late and roaming usage will keep arriving. Separate the number from the relationship: other lines on the account keep working, shared allowances stay intact for remaining members, and any authentication or notification path using the number moves before cutover.
What is the difference between suspend, bar, cease and terminate, and why do you care?
They are distinct states with distinct network effects, billing effects and reversibility. A bar restricts what the subscriber may do while the subscription stays live and usually still charges rental; barring for non-payment is reversible on payment and must be reversible quickly. A suspend is broader and often customer-requested, sometimes with reduced rental. A cease ends the service: network configuration removed, resources released, recurring charges stopped prorated to the date, final bill prepared. Termination of the account or contract is broader still and pulls in early termination charges, equipment debt and instalment plans.
You care because the state determines four things independently, and a system modelling a single active flag cannot express the combination in the third row.
| State | Network carries traffic | Recurring charges | Usage still rateable | Resources reclaimable |
|---|---|---|---|---|
| Barred | Restricted | Continue | Yes | No |
| Suspended | No | Often reduced | Yes | No |
| Ceased | No | Stop, prorated | Yes, for late arrivals | Only after quarantine |
| Terminated | No | Stopped | Closed | Yes |
That third row is the one to say out loud: a ceased subscription is unserviceable and still rateable, because roaming and late usage keep arriving after the network has stopped carrying anything.
Mediation, rating and charging
What does mediation do that rating cannot do for itself?
It turns element output into commercially meaningful events, and every stage of that is work rating has no business doing: collection from many element types over many transports, decoding of element-specific and frequently binary formats, normalisation into a common shape, deduplication because elements retry and files get delivered twice, correlation of several records describing one event into one, enrichment with facts the element did not have such as which subscription an identity belongs to, filtering of what is not billable, and distribution to the several consumers wanting the same usage for different purposes.
Keep it separate because all of that changes when the network changes, while rating changes when the commercial model changes. Put decoding inside rating and a new element type becomes a rating release, which means a network project now waits on a billing release window. That is the same rate-of-change argument that justifies the OSS and BSS boundary, applied one level down, and making that connection explicit is worth credit because it shows the principle is load-bearing rather than recited.
Usage arrives duplicated, late and out of order, and the tariff changed mid-month. How does that work?
Two mechanisms, and keeping them separate is the answer. Deduplication needs a persisted key window sized to the worst lateness you actually observe rather than the lateness you hope for, which means measuring it and revisiting the number, and it means the key derives from fields the element guarantees stable rather than from an arrival timestamp.
Pricing needs event time, not arrival time. Rating prices each record against the tariff version effective when the usage occurred, which requires effective-dated versions and requires the rated output to record which version priced it, so "which version priced this call" is a lookup six weeks later rather than an argument. Out-of-order arrival then stops mattering for postpaid pricing, though it still matters for anything cumulative such as an allowance or a tiered rate, which is why those are computed against a counter that can be recomputed rather than only incremented. The detailed treatment works through the interaction with prepaid, where you cannot wait.
Is prepaid just postpaid with lower latency?
No, and this is a reliable level-setting question. Postpaid rating is a valuation of something that already happened: the record arrives, is priced, accumulates, and a cycle eventually closes. Nothing had to be decided before the usage occurred.
Online charging is an authorisation of something about to happen. The network asks permission, the charging system reserves quota against a live balance, the network consumes and reports, and the reservation is settled with the difference returned. That imposes requirements postpaid does not have: the balance is authoritative and must be consistent; quota granularity trades signalling volume against the amount of unbillable usage on a failure; the network's behaviour when charging is unreachable is a policy decision with revenue consequences one way and customer consequences the other; and an abandoned session leaves a reservation that must be released by a timeout rather than by a message you will never receive.
| Dimension | Offline rating for postpaid | Online charging for prepaid |
|---|---|---|
| When the decision is made | After the usage, from a record | Before or during the usage |
| Authority over the balance | Accumulates a total, no gate | The balance is the gate |
| Effect of late records | Charge lands on a later bill | Usage already happened unpriced |
| Effect of unreachability | Records queue and wait | The network must allow or deny on policy |
| Abandoned session | Nothing to release | A reservation released by timeout |
Convergent charging means one system doing both against one balance and one catalogue, which removes a class of divergence and concentrates the availability requirement into one component. State that trade rather than presenting convergence as unambiguously good, because the failure of a convergent system is broader than the failure of either half.
A family plan shares 100 GB across five SIMs, each capped at 30 GB. How does charging enforce both?
Every reservation must decrement the shared pool and the member's own counter as one atomic act, granting the smaller of the two remainders. That has a structural consequence worth naming: the serialisation point becomes the group rather than the subscriber, so five devices in one household contend on one counter, and the design has to survive that contention rather than assume per-subscriber independence.
It also has a customer-visible consequence a good answer volunteers: a member with headroom under their own cap can still be refused because the pool is gone, and the message the customer receives has to say which limit was hit or the contact centre receives the call instead. The worked version also covers what a partial grant does to the reported balance.
A subscriber roams and the charges arrive days later. What does that break?
Roaming splits the usage from the record of it. The visited operator meters the usage and sends files on an agreed schedule and format, so your systems learn about it after the fact. Postpaid can rate late records, and the consequence is a record that may miss the cycle it belongs to, meaning either a late charge on the following bill or a deliberate wait before closing. Prepaid cannot wait at all, because the point of a balance is enforcement before the money is spent, so operators use near-real-time exchange mechanisms to shorten the window and accept residual exposure.
The design question underneath is which usage you allow before you have proof of it, and the answer is a risk position rather than a technical one: caps on unverified roaming usage, monitoring of accumulating exposure, and acceptance that the residual is a fraud cost. A compromised account roaming on expensive destinations accumulates loss for exactly as long as your visibility lags. The longer treatment works through what the delay costs.
Billing and collections
Take me through a bill cycle.
A cycle closes a period for a group of accounts. Inputs are frozen at cut-off so the run is reproducible. Recurring charges are assembled for the period, including prorated amounts from mid-cycle changes. Rated usage for the period is collected. One-off charges and adjustments are added. Discounts and bundle allowances are applied, in an order that has to be defined because discount interaction is not commutative and two implementations of "apply the best discount" will disagree. Tax is calculated on the result. An invoice is produced and issued, the receivable is posted to the ledger, and the document is immutable from that moment. Payment is expected, and where it does not arrive, collections engages.
Three properties are worth naming because the follow-up questions all depend on them. The run is a sequence of per-account commitments rather than one transaction. Charge assembly must be deterministic from the frozen inputs, so the same account billed twice from the same inputs produces the same invoice. And an invoice is never edited after issue because it is a legal document; corrections are credit notes and adjustments appearing on a later invoice.
The bill run dies two thirds of the way through and the cycle closes tomorrow. What do you do?
Restart rather than rerun, and the distinction is the whole answer. Because the run is a sequence of per-account commitments rather than one transaction, each account must be either fully billed or untouched, and a restart resumes at the first uncommitted account. That only works if per-account completion is durable and charge assembly is deterministic from inputs frozen at cut-off, so an account billed after the restart produces the invoice it would have produced before it.
Then triage rather than block. If a small number of accounts are causing the failure, quarantine them, complete the cycle for everyone else, and bill the quarantined set separately once diagnosed, because missing a cycle for the whole base is far worse than a delayed invoice for a hundred accounts. The scenario in full covers what you must not do, which is rerun from the start over accounts already committed.
A customer changed plan mid-cycle and disputes the bill. What are they looking at?
Typically a credit for the unused portion of advance rental on the old plan, a prorated charge for the remainder of the period on the new plan, possibly a one-off change fee, and usage priced under whichever plan was effective when each event occurred. Four lines where the customer expected one, in a month whose total is unlike any other, which is why this is the highest-volume billing dispute there is.
The graded content is what you do about it rather than the arithmetic. Proration must be computed on a stated basis, and the same basis everywhere, because a daily rate on actual days in the month and one on a thirty-day convention give different answers and both will be defended by whoever implemented them. The invoice has to present the lines so a contact centre agent can narrate them. And aligning a plan change with the cycle boundary where the product allows removes the proration lines entirely, which removes most of the contact volume. The cycle and proration walkthrough goes into where leakage hides in those lines.
A customer was barred for non-payment and they paid three weeks ago. Explain.
Almost always the money arrived and was never allocated to the invoice dunning was watching. The account balance is zero, the specific document is still open, and dunning ages open documents rather than balances, so the subscriber crossed a threshold and was barred while owing nothing. The usual causes are a payment reference that did not match, a payment received against the account rather than the document, a partial payment allocated to the wrong document, a credit note offsetting a different invoice, or a direct debit posted to a suspense account.
The response has three timescales and keeping them apart is the signal. Unbar now, because the customer is out of service and the fault is yours. Correct the allocation today, so the document closes and the next dunning pass does not re-bar them, which is a specific and common second incident. Then change what dunning reads, so the state it acts on is one a human can defend: unallocated cash on the account should suppress or delay dunning, and the same applies to open disputes and pending refunds.
Revenue assurance and leakage
Where does revenue leak in an estate like this?
At every boundary where two systems can disagree, which is why the answer is a list of reconciliations rather than a list of bugs.
| Reconciliation | What it catches |
|---|---|
| Usage on the network against usage rated | Mediation dropping files, a decoder failing silently on a new record type, filters rejecting billable events |
| Services active against subscriptions billed | Half-completed activations one way, half-completed ceases the other |
| Orders raised against services delivered | Fallout that nobody worked |
| Subscriptions against the catalogue versions they reference | Subscribers left on withdrawn offerings and prices that drifted |
| Discounts applied against discounts authorised | Configuration error, and deliberate misuse |
| Partner traffic recorded against partner settlement | Interconnect and content revenue going unclaimed |
The framing that earns credit is that leakage is a symptom of missing ownership rather than of carelessness. Every row above exists because a fact is held in two places, and the durable fix is usually to reduce that to one place, though you run the reconciliation forever anyway. Note also that the second row is the only one most operators run in both directions, and that the two directions have different owners: unbilled service is a finance problem, and billed-but-absent service is a customer problem that will find you without help.
How would you find every subscriber currently in a bad state, and how often would you look?
Define the state as a difference between two populations you can enumerate independently, which forces you to name the identifier joining them and to discover, often uncomfortably, that there is not one. Then set the cadence from how fast the state accumulates and how much each instance costs: a divergence costing rental every month and appearing a few times a day is a daily job, and a compliance divergence is a daily job regardless of volume because its cost is not proportional to its count.
What makes it real rather than a report is that every non-empty difference goes to a queue with an owner and an expected resolution time, that findings are classified rather than counted so the generator can be identified, and that the job keeps history so the rate is visible. The rate matters more than the backlog. One divergence found by chance implies a rate, and a rate implies either a defective task, a partner that acknowledges without completing, or a manual process routinely bypassing the workflow, and those three have completely different fixes.
Would you ever auto-remediate a divergence without a human?
Yes, for classes where the correct action is unambiguous, the input is trustworthy and the blast radius of being wrong is bounded. Re-driving a specific idempotent task on an order stalled at a known step, with an attempt limit and a fallout route after it, is a reasonable candidate. So is releasing a resource reserved by an order that was cancelled and has passed a defined age, because the reservation has no claimant.
Refuse the classes where remediation creates a commercial fact. Creating a subscription from observed network state bills someone for something they never ordered; deprovisioning from an absent commercial record disconnects a customer whose record was lost rather than never created. The rule to state, because it generalises past this question, is that automation may converge the estate towards recorded intent and may never invent the intent.
Finance want a leakage number by Friday. What do you give them?
A bounded number with its method attached, and an explicit statement of what it excludes. The instinct to produce a single total is what gets people into trouble, because a leakage figure without a method is unfalsifiable and will be quoted back at you for two years.
So give the reconciliations you can run now with the identifiers that exist, quantified with the assumptions visible, and separately list the boundaries you cannot currently reconcile and what it would take to be able to. That second list is more valuable than the number, because it converts an audit finding into a work plan. And distinguish leakage, which is revenue you were entitled to and did not collect, from uncollectible revenue, which is money you billed and will not receive, and from discounting, which is money you chose not to charge. Mixing those three produces a number nobody can act on.
You have inherited an estate with no reconciliation at all. What do you do first?
Pick the reconciliation with the highest expected value per week of effort, which is usually services active on the network against subscriptions billed, because it runs in both directions, it finds both leakage and overcharging, and both directions have an owner who will act on a finding. Establish the identifier joining the two populations before promising anything, because if there is not one then building it is the first piece of work and the reconciliation is the second.
Then resist the pull towards a platform. A single query on a schedule, whose output goes to a named person with an expected resolution time, produces repairs. A reconciliation programme with a tool selection produces a roadmap. Once one reconciliation is running and findings are being closed, the second is far easier to fund, because you now have a rate and a recovered amount to argue with rather than a proposal.
Network inventory and serviceability
How would you model physical and logical network resources?
As separate layers linked by assignment, because they change independently: physical equipment driven by engineering work, logical resources riding on it driven by configuration, and services consuming those driven by customer orders. It is the assignment rather than the resource that carries the customer relationship.
Then insist on a real lifecycle rather than a boolean. A resource can be planned, built, available, reserved, assigned, faulty, quarantined or being decommissioned, and several are not substitutable. Reserved and assigned differ by whether an order completed, which determines whether a timeout may reclaim it. Quarantined and available differ by whether reuse is safe yet, which is a different question from whether the resource works. And planned differs from built by whether an order may be accepted against it today, which is the state that lets you sell for a future date without lying about a present one.
Discovery disagrees with the inventory. Which one is right?
Neither automatically, and answering "the network, obviously" is the mistake the question hunts for. The inventory owns intent: what should be there, who it belongs to, what was ordered. Discovery observes state: what a device reports right now. A difference between them is a classified fault rather than an error in the record.
So classify. Something in the network with no record is an undocumented change, a lab configuration on a production element, or a record that was lost, each with a different resolution. Something in the record with nothing in the network is a failed activation, a service ceased on the network only, or a discovery gap where the device does not report that object type at all. Something in both with different attributes is commonest and most interesting, usually a manual change that solved a real problem. Overwriting the inventory from discovery destroys the intent, which is the only thing telling you what to restore after a device is replaced. The worked answer covers the reconciliation loop in more detail.
Sales keep selling broadband at addresses we cannot serve. What is wrong?
Serviceability is four different questions and a single yes-or-no answer collapses them. Is the address real and resolvable to a canonical identity. Does the network pass it. Is there spare capacity on the specific piece of infrastructure that would serve it. And may we sell there, which is a commercial and regulatory question about wholesale agreements, launch phasing and obligations.
Most failures start in the address model rather than the network model, and that is the part earning credit. The customer types an address, the channel matches it loosely, and the match resolves to a neighbouring property or to a building rather than a unit within it, so the capacity answer is correct for somewhere else. Fixing that means one canonical address authority with a stable identifier every system carries, matching that returns candidates rather than a best guess, and an explicit unmatched path rather than a silent nearest hit. The full breakdown separates the four questions and says which system should own each.
A planned upgrade window opens and correlation stops working. Why?
Because correlation is a traversal of the inventory graph, and an upgrade changes exactly the part being traversed. Root-cause analysis walks from an alarming resource up its dependency chain to a common ancestor; when the chain is being rebuilt, the traversal fails or returns the wrong ancestor, and thousands of alarms arrive uncorrelated.
The remedies all involve modelling the change as data. Model planned state as well as as-built state, so the graph knows what the topology is about to become. Carry the set of resources a change touches as data on the change record rather than as free text in a ticket. Scope suppression to that resource set rather than to a time window, because a window suppresses genuine unrelated faults and misses the ones caused by the change spilling outside its planned scope. And confirm recovery positively rather than assuming the end of the window means the end of the impact. The detailed answer works through the interaction between change data and suppression.
Network services and technology change
What do slicing and edge computing change for the systems that sell and provision?
They turn a service into a parameterised, capacity-constrained thing with a location, and each of those three words breaks an assumption. Parameterised means the catalogue must model performance characteristics as typed attributes with units rather than as marketing text, and the order journey must collect them. Capacity-constrained means the order flow gains a qualification and reservation step that can legitimately say no, which most consumer-oriented order flows have never had to express. Located means the service binds to a site or area, so the address and site model becomes part of the commercial product rather than a delivery detail.
The consequence senior candidates reach and others do not is that a performance commitment sold as a product characteristic has to be measured and fed back into billing and into service credits, so assurance data becomes a billing input. That is a new coupling between OSS and BSS the traditional boundary did not carry, and it deserves to be designed deliberately rather than discovered. The fuller treatment works through the catalogue and ordering changes.
The 3G network is being switched off in eighteen months. What work does that create?
Mostly the work of identifying who loses service and moving them, and the point to make immediately is that network data alone cannot tell you who they are. A subscriber not registering on a newer radio technology might have an incapable device, or a capable device with a subscription or provisioning gap, or might simply live where the newer coverage is poor. Device capability comes from the device identity reported by the network rather than from anything the customer told you, and matching that against a device database is how the population is established.
Then the estate work. Voice over the packet network needs both a subscription entitlement and provisioning in the relevant network functions, in bulk, for a large base, which is a mass-change programme rather than a per-order one. The catalogue may carry offerings whose technical realisation is being removed. And the population no marketing campaign can reach is the machine-to-machine base: devices in meters, alarms, vehicles and industrial equipment, with no user, sometimes no known owner, sometimes physically inaccessible, which is what turns a migration into a multi-year negotiation. The full answer covers the identification problem in depth.
You are pushing a configuration change into the mobile core tonight. How do you avoid taking the network down?
Start by rejecting the premise that redundancy protects you, because blast radius is a property of the change process rather than of the topology diagram. Identical configuration applied to both halves of a resilient pair fails both halves, and that is the mechanism behind a large share of national outages.
So stage the exposure: one element, then a subset, then the estate, with soak time between stages long enough for the failure mode to appear, which for subscriber-affecting changes means long enough for real traffic patterns to cycle. Health checks must be subscriber-visible rather than element-visible, because an element reporting healthy while rejecting registrations is the specific case staging exists to catch. The management path must survive the outage you are risking, since a change that removes your own access removes your rollback. And the rollback must have been executed rather than written down, because an untested rollback is a plan and not a control. The detailed version covers soak time and health-check selection.
Testing, release and judgement
No environment holds the whole stack. How do you get assurance?
Accept it and decompose the assurance question the way you decomposed the estate. Every boundary gets an explicit versioned contract covering request and response shapes, field meanings, error taxonomy, retry and idempotency semantics, timeout behaviour and event ordering. The test that makes the contract real is executable by each side alone: the consumer runs its suite against a stub built from the contract, the provider runs the same suite against itself, so nobody needs the other party's environment to be up.
Two boundary behaviours deserve explicit tests because they cause most fallout and are almost never covered: the duplicate request, because impatient retries are guaranteed, and the partial or timed-out response, where the consumer must not conclude failure while the provider has in fact acted. In an order path those produce double activations and orphaned services respectively.
Then simulate the network and be explicit about what the simulator does not reproduce: timing under load, the element that accepts a command and applies it minutes later, sequence-dependent hardware behaviour, and the command that succeeds while the service still does not work. Those faults reach production, so the simulator's job is to make everything else cheap enough to test continuously.
And test data?
Say it is the harder half, because teams that get their environment discover the data was the constraint. An end-to-end order needs a coherent subscriber across every system in the chain, and that coherence is what nobody owns. Identifiers must line up across CRM, charging, billing and the network, with number ranges reserved so a test order cannot activate a real subscriber's identity. Lifecycle states must be reachable: a subscriber mid-contract, one in arrears, one suspended, one with an in-flight change, and manufacturing those by hand takes longer than the test. Refreshing from production drags in personal data, and naive masking destroys the cross-system joins that made the copy useful, so masking must be deterministic and applied across every system in one pass.
Treat test data as a product with an owner: a generator that creates a consistent customer across the estate on request, a small library of curated hard cases, and a reset mechanism. That is usually what converts a six-week integration test into a two-day one.
What assurance do you get from production itself?
The only place the whole stack exists is production, so design to get assurance from it safely. The main mechanism is a continuous synthetic journey: on a schedule, place a real order for a test subscriber on reserved identifiers, drive it to activation, verify service on the network, rate a piece of usage, produce and check the resulting charge, then tear it down. Run it often and alert on failure or slowdown, and you have an honest measure of whether the estate works end to end, one that catches the vendor patch nobody mentioned, the certificate that expired and the queue that silently stopped.
Build it with production-grade care, and name the specific hazard: those test subscribers must never enter revenue reports, regulatory returns, marketing lists or capacity forecasts. Around it sit the release techniques limiting what a bad change can do: cohort release to real subscribers, feature switches separating deploying from enabling, and a rehearsed rollback.
Two vendors each pass their own tests and the journey still fails. Where do you look?
At the contract rather than at either implementation, because that is what the symptom is telling you. Both sides satisfy their reading of the interface, so the readings differ, and the difference is almost always in something the contract left implicit: an optional field one side always sends and the other ignores, a code list one side has extended, an ordering assumption about events, a timeout mismatch where the consumer gives up before the provider replies, a character encoding, a time zone, or a retry the provider treats as a new request.
The process answer matters as much as the diagnosis. Capture the actual exchange at the boundary rather than reasoning from logs on either side, because each side's log records what it believed it sent, and both logs will look correct. Then encode the resolution as a test in the shared suite, because the same ambiguity reappears at the next release otherwise, and a fix that lives only in a ticket has not changed the contract.
A vendor says the behaviour you are seeing is by design and will not change it. Now what?
Establish what the contract says, because that determines whether this is a defect conversation or a change request, and the two are handled by different people. If the behaviour contradicts the agreed interface it is a defect, and the route is contractual as well as technical. If the interface never specified it, the vendor is right and you have found a gap in your own specification rather than in their product.
Either way, design around it in a place you control and make the workaround visible. An adapter that normalises the behaviour at the boundary is better than the behaviour leaking into three consumers, and a documented compensating control is better than an undocumented assumption. Then put it on the register to revisit at the next release, because workarounds that nobody records become the reason nobody can upgrade two years later.
How do you decide what not to do when everything is urgent?
Rank by irreversibility and by who else is affected, not by who is loudest. A regulatory deadline and a bill cycle date are fixed and their consequences are external to your organisation. An activation defect generating divergences accumulates cost every day it runs and gets more expensive to repair as the population grows. A launch date is negotiable more often than it is presented as being. A refactor is almost always deferrable, and saying so protects your credibility for the occasions when it is not.
Then make the trade visible rather than absorbing it. Stating in writing that a piece of work will not happen this quarter, with the consequence named, is the behaviour that separates people who are trusted with scope from people who are given tasks. Interviewers for senior roles ask this specifically to find out which of those they are talking to.
Marketing want a launch in three weeks and fulfilment needs six. What do you do?
Separate what can be recombined from what needs new capability, then offer a reduced first launch rather than a manual workaround presented as automation. A limited channel, geography or volume with a known manual step is defensible if the step is funded, staffed and time-boxed with a date at which it is removed. What is not defensible is agreeing to the date and relying on an operations team absorbing the difference indefinitely, because that becomes permanent and invisible.
How to prepare
Learn the order-to-activate spine well enough to draw it from memory and name what each step owns, because a large fraction of questions here are a probe into one point on it. Then pick the two or three domains closest to the role and go deep enough to hold opinions: what you would do differently, what you have seen fail, which trade-off you would take.
Rehearse the scenario answers out loud, because the structure is what is graded and structure is easy to lose under pressure: bound the symptom, separate restoring service from fixing the cause from fixing the class, name which failures are silent and how you would detect them, and say where the decision stops being yours.
Then bring one estate you know in detail, even if it is small and even if it went badly. A candidate who can say what an identifier mismatch between two particular systems cost them is more convincing than any amount of framework fluency. Be honest about the parts you have not touched, because an interviewer who catches you overstating will discount everything else you said.