A fill arrives for an order your system has already marked cancelled. What do you do?
Book the fill. The venue decides what happened, so a fill against a cancelled order means your state was wrong, and the position, risk headroom and client record all have to be corrected from the venue's version. The design fix is to make cancelled a state only the venue can grant.
What the interviewer is scoring
- Does the candidate treat the venue as the authority rather than looking for a way to reject the fill
- Whether the two cases are separated, a genuine cancel race against a fill arriving after a confirmed cancel
- That deduplication is described by execution identifier rather than by arrival order or by trusting the session
- Whether the candidate follows the consequence into risk headroom that was released on the cancel
- Does the answer say what the client was shown, and who carries the position if they were told wrongly
Answer
Start from who is allowed to decide
An order exists at the venue, not in your database. Your record is a projection of the venue's state, kept current by messages that can be delayed, duplicated and interleaved. So the first move is not to work out how to reject the fill, because you cannot: a fill is a report that a trade has happened, and a trade that has happened is an obligation that will be cleared and settled whatever your order state says. The only question is which of your records is wrong and how far the correction has to travel.
That reframing is most of what is being scored. Candidates who reach for "we should reject it" or "we should ignore it as a duplicate" have made your system the authority over a fact it does not own.
Two situations that look identical in the log
Separate them before doing anything else, because the remedies differ.
The first is a genuine race. You sent a cancel request and your system marked the order cancelled on sending rather than on confirmation. Meanwhile an aggressor arrived at the venue and matched against the resting order. The venue processed the match first, so the cancel is answered with a cancel reject carrying a reason of too late to cancel, and the fill is real. Nothing is broken except your optimism: the order was live throughout, and the state you displayed was a guess.
The second is worse. The venue confirmed the cancel, and a fill arrives afterwards. Now one of a small number of things is true. It may be a retransmitted or resent message you have already applied, in which case the correction is deduplication rather than booking. It may belong to a different order in a cancel-replace chain, where a fill referencing the superseded client order ID is read as a fill on an order you believe is gone. It may have been generated before the cancel and delivered after it, in which case your handler has applied messages in arrival order instead of by their own semantics. Or the venue has genuinely erred, which happens, and is resolved by the venue busting the trade rather than by you deciding to disregard it.
sequenceDiagram
participant C as Client
participant O as Broker OMS
participant V as Venue
C->>O: Cancel request
O->>V: Cancel forwarded
V->>V: Aggressor matches first
V-->>O: Execution report - fill
V-->>O: Cancel reject - too late to cancel
O-->>C: Fill, and the order was never cancelledLook at the ordering at the venue rather than on the wire: the two responses are generated in the order the venue processed the events, and your handler must reach the same conclusion regardless of which one it reads first.
Applying the message safely
The rule that makes both cases tractable is that execution reports are applied by identity, not by arrival. Every fill carries its own execution identifier, and a correction or cancellation of a fill references the identifier of the fill it acts on. Keep a set of applied identifiers per order and the duplicate problem disappears without any reasoning about order state.
// The fill is a fact. The order state is a cache of the venue's opinion.
// So position moves first, unconditionally, and order state is repaired after.
void onFill(ExecutionReport er, Order order) {
if (!order.applyOnce(er.execId())) return; // idempotent: replays are free
position.apply(er.symbol(), er.side(), er.lastQty(), er.lastPx());
order.cumQty(er.cumQty()); // venue's running totals win
order.leavesQty(er.leavesQty()); // over anything we computed
order.status(er.ordStatus());
// Cancelling released headroom that this fill has now consumed. The
// reservation must be re-taken even if that pushes the account over its
// limit, because refusing to record a trade does not undo the trade.
limits.consumeAfterUnexpectedFill(order.account(), er.lastQty(), er.lastPx());
if (er.leavesQty() > 0) order.reopenIfLocallyTerminal();
}
Two details in that sketch matter more than the shape of it. LeavesQty and CumQty from the venue replace whatever you had computed rather than being reconciled against it, because a divergence means your arithmetic is behind, not that the venue is wrong. And a locally terminal order has to be capable of returning to a working state, which is a constraint most naive state machines fail: if cancelled is modelled as absorbing, a late fill leaves the position moving under an order that cannot be updated to explain it.
The consequence people forget is in risk, not in the order
A cancel usually releases limit headroom. That is correct behaviour, and it is exactly what makes the late fill dangerous. Between the optimistic cancel and the fill, the desk may have used the freed headroom for another order. When the fill lands you have consumed the same headroom twice, and the account can be over its limit through no rule being broken.
The honest design accepts that. Limit consumption on an unexpected fill must always succeed and may take the account negative, with an immediate alert and a block on new orders for that account until it is back inside. A system that refuses the reservation and therefore refuses to record the fill has chosen a clean limit report over a correct position, which is the wrong trade in every direction, including the regulatory one.
What the client was told is a separate loss
If the order was only pending cancel and you displayed it as cancelled, the client acted on a state that did not exist. A trader who believed the order was gone may have replaced it elsewhere and is now long twice. The position is real either way; the argument is only about who owns it, and firms settle that argument out of an error account rather than out of the order book.
This is why the display question deserves a decision rather than a default. Pending cancel is a real, showable state, and showing it costs nothing except the appearance of decisiveness. Optimistically showing cancelled makes the common path look instant and makes the uncommon path a client dispute with your own log as the evidence against you.
Where the strong answer separates
The adequate answer books the fill and updates the position. The strong one goes on to say how you would know if this had happened silently, because the same race with a lost or discarded message produces no visible fill and a position you do not know you have. The answer is that the venue's drop copy and its end-of-day trade file are the reconciliation authority, not your own message log, and the control is a daily comparison of executions rather than an alert on the anomaly you happened to think of. An order state machine that can only be corrected when a surprising message arrives is not reconciled; it is lucky.
Your order record is a claim about somebody else's system. Design every state transition so that being wrong is recoverable, and reconcile against the venue rather than against your own history.
Likely follow-ups
- Where does the loss sit if you told the client the order was cancelled and they replaced it on another venue?
- How would you model an amend so that a failed replace cannot leave you with neither the old order nor the new one?
- Which of your order states are safe to display optimistically, and which must wait for the venue?
- How do you prove at end of day that your fills and the venue's trade file agree, and what do you do about a difference found after the cut-off?
Related questions
- The MES is unreachable for four hours and the line keeps producing. What happens to the production record, and how do you reconcile afterwards?hardAlso on reconciliation and idempotency5 min
- A subscriber's service is working on the network but no subscription exists in billing. How did that happen, and what do you do about it?hardAlso on reconciliation and idempotency5 min
- A payment API times out and the client retries. How do you guarantee the customer is not charged twice?hardAlso on idempotency and reconciliation6 min
- Walk me through an order's journey from placement to delivery. How would you model its status?hardAlso on idempotency and order-lifecycle6 min
- The monthly bill run dies two thirds of the way through and the cycle closes tomorrow. What do you do?hardAlso on idempotency5 min
- How would you design a data pipeline you can safely re-run?hardAlso on idempotency6 min
- You own the API test suite for a service from scratch. How do you structure it, and what do you mock?hardAlso on idempotency7 min
- An event for 23:58 yesterday arrives at 00:20 today, after you have already published yesterday's totals. What should your pipeline do?hardAlso on idempotency4 min