You have to release a change to the order-routing path. Walk me through the release, and tell me what makes it different from deploying any other service.

The gateway holds obligations, so you cannot drain it

A stateless web service is deployed by taking a node out of the load balancer, letting its work finish, and replacing it. An order gateway cannot be treated that way, because the work does not finish. Orders it sent are resting at a venue with the venue's own identifiers, and those orders will be matched whether or not the process that sent them is running. The session carrying execution reports back has monotonic sequence numbers that both sides track. And the mapping from a venue order back to the parent instruction it belongs to is state that has to survive the restart, or the fills that arrive afterwards belong to nothing.

So the first question in the release plan is not how to roll the fleet but what happens to the live obligations of the node you are about to stop. There are only a few honest answers. Cancel everything that node has resting, accept the cost, and restart clean. Restart with the session and the order book recovered from durable state and reconciled against the venue before the node is allowed to send anything new. Or do not deploy while the market is open. Firms choose all three depending on the venue and the product, and a candidate who has not noticed that the choice exists has not worked on one of these systems.

Two versions of the truth on one fleet

A rolling release means that for some window, part of the fleet behaves one way and part behaves another, while both are transacting. That is tolerable if the two versions agree about everything they share and intolerable otherwise, and the things they share are easy to overlook: the interpretation of a configuration flag, the encoding of a client order identifier, the durable order store's schema, the definition of a risk limit and the counters it is checked against.

The subtlest of these is configuration. A flag whose meaning has changed between versions means the two halves of the fleet read the same value and do different things, and nothing in the deployment will report an error. This is why configuration compatibility has to be reasoned about with the same seriousness as message compatibility: either the flag is new, or its meaning is unchanged, or the deployment is not rolling.

The related discipline is that code you no longer want executed must be removed rather than disabled. A path retained but switched off is a path that can be switched on, by a flag that gets reused, a default that changes, or a config file that is deployed to one host and not another. Deleted code cannot run. Disabled code is a latent instruction waiting for somebody to give it the wrong value.

Verification has to be about positions, not liveness

The instinct after a release is to check that the process is up, the session is connected, and the error rate is flat. All three can be perfectly healthy while the system does the wrong thing, because sending orders is the normal behaviour of an order gateway and the venue will accept every one of them.

The check that actually detects this class of failure is a reconciliation, running continuously, between what the gateway believes it has working and what the venue believes it holds. Every venue order maps to exactly one parent instruction; the sum of children never exceeds the parent's quantity; the position implied by fills matches the position the risk system holds; and the count of live orders per session matches the venue's view. Drop copies exist precisely so that this comparison uses a path independent of the one that might be broken. Anything that trades without a parent it can be attributed to is the single highest-severity signal such a system can produce, and it should stop the flow rather than raise a ticket.

flowchart TD
    A[Release candidate] --> B[Venue conformance run out of hours]
    B --> C[Shadow run against drop copy]
    C --> D[Deploy with fleet homogeneity enforced]
    D --> E[Continuous parent-to-child reconciliation]
    E --> F{Unattributed order or position break}
    F -->|Yes| G[Scoped kill switch and manual close-out]
    F -->|No| H[Ramp flow]

The step people leave out is the third one, enforcing homogeneity: a deployment tool that reports success when it configured most of the fleet has told you nothing about the host that matters.

Rollback does not exist in the sense you are used to

Reverting the binary stops the system doing the wrong thing again. It does not unwind the trades already executed, because those are obligations to counterparties and will clear and settle regardless. The remedy for the position is a market transaction, at the prices available when you discover the problem, and the loss lands in an error account.

That asymmetry should change the release plan rather than merely be acknowledged. It argues for verifying against the venue in a conformance environment before market hours, for shadow-running the new path against a drop copy so its decisions can be compared with the incumbent's without sending anything, and for ramping flow deliberately once live rather than switching all of it at once. Progressive delivery in a trading system is not a percentage of users; it is a small number of instruments, a small notional cap and a person watching.

A kill switch somebody can actually pull

Every firm has a kill switch and the useful question is what it is scoped to. "Stop trading" is not an operable instruction at three in the afternoon. What an operator needs is the ability to stop something they can name from the evidence in front of them: this session, this gateway, this strategy, this account, this instrument. The scope has to match the granularity of the alert, because the person acting will be reasoning from an alert.

Two properties matter beyond scope. The switch must be reachable when the component that would decide to pull it has failed, which means it cannot live inside the same process or depend on the same store. And its effect on existing state has to be defined and known in advance: whether it merely blocks new orders, cancels resting ones, or both. An operator discovering during an incident that the switch stopped new orders and left a thousand resting is discovering it at the worst possible time.

What Knight Capital established about partial deployments

In August 2012, Knight Capital deployed new code to seven of its eight order-routing servers. On the eighth, an old code path was reactivated by a flag that had been repurposed, and that server began sending orders into the market that nothing was tracking against a parent instruction. Roughly $440M was lost in about forty-five minutes, and during it staff could not identify which component to stop.

Every element of the release discipline above maps onto one of those facts. The deployment was not verified as complete, so the fleet was not homogeneous. The old path had been retained rather than deleted, so it remained runnable. The flag's meaning had changed, so the same configuration value meant two different things on two versions. Nothing reconciled orders against parent intent in real time, so the failure was visible only as a market event. And the kill switch was not scoped to anything anyone could name under pressure, so the fastest available remedy was minutes of diagnosis rather than seconds of action. The interviewer asking this question is checking whether you have internalised those five, or only the anecdote.

An order gateway deployment is a change to a system that is currently holding obligations to counterparties. Nothing about it can be verified by checking that the process is healthy, and nothing about it can be undone by putting the old binary back.