You get paged, the service is down, and nobody else is online yet. How do you run the incident?
Declare it, take the incident commander role explicitly, and mitigate before you diagnose - roll back or shed load first, understand later. Keep one channel as the record, publish updates on a fixed cadence, hand over rather than push through, and let the review produce owned actions instead of narrative.
What the interviewer is scoring
- Does the candidate mitigate before diagnosing, and can they say why that order is not laziness
- Whether the commander role is described as separate from doing the hands-on work
- That declaring early is treated as cheap and delaying as expensive
- Whether stakeholder communication happens on a cadence rather than when someone asks
- Can they distinguish a review that produces owned actions from one that produces a narrative
Answer
Declare, then stop the bleeding
The first decision is to say out loud that this is an incident. Declaring is cheap — a channel, a page to a second responder, a status entry — and not declaring is expensive, because everything that follows depends on other people knowing this is live. Teams consistently under-declare out of a worry about looking dramatic, and the cost lands twenty minutes later when support is answering customers from guesswork.
The second decision is to fix the symptom before understanding the cause. If a deploy went out in the last hour, roll it back. If a feature flag turned something on, turn it off. If one dependency is the source, shed the traffic that touches it. Understanding is the more satisfying activity and it is the wrong one to be doing while the error rate is at forty percent, because the diagnosis is equally available afterwards and the outage minutes are not. The exception is narrow and worth naming: when the mitigation could plausibly make things worse — a failover of unknown safety, a destructive rollback across a schema change — you spend a bounded few minutes establishing that it will not, and you say the bound out loud.
Hold the incident, or work it, but know which
The reason incident command exists as a named role is that the person with their hands on the terminal cannot also track time, decide priorities and talk to the business. Attention does not divide that way, and the failure is predictable: the investigation goes deep on the first plausible theory while nobody notices that thirty minutes passed and no update went out.
So as first responder you are the commander by default, and your job is to keep a current picture of impact, decide what is tried next, and delegate the doing as people arrive. The moment a second engineer joins, hand the hands-on work to them if you have been the one typing. If the incident is small, one person holds both roles knowingly — that is fine, and different from holding both without noticing.
The channel is the record
Everything material goes into one place, in the open, as it happens: what you observed, what you tried, what it did, what you decided and what you ruled out. This is not bureaucracy during an outage, it serves three immediate purposes. Someone joining at minute twenty reads themselves in without interrupting anyone. Two people stop independently trying the same restart. And the timeline the review will need already exists, written while memory was accurate rather than reconstructed a week later when everyone remembers the order of events wrongly.
Write timestamps and write the negative results, because "we confirmed the database was healthy at 02:14" is what stops the next responder repeating that check.
Comms on a clock
Stakeholders will interrupt the response to ask for status unless status arrives before they need to ask, so commit to a cadence — every fifteen or thirty minutes for a major incident — and publish even when there is nothing new, because "still investigating, no change, next update at 03:15" is information. Keep it to impact, scope, what is being done and when you will speak next. Do not put causes in a customer-facing update while you are still guessing at them; a retracted cause costs more trust than a slower answer.
02:47 UTC - Checkout unavailable for all customers since 02:31.
Cause under investigation. Rollback of the 02:28 release in progress.
Next update 03:00 UTC.
Hand over rather than push through
Judgement degrades quickly, and the responder who has been on it for four hours is the last person who will notice their own degradation. A handover is a deliberate transfer: current impact, timeline so far, theories still open, theories eliminated, what is running right now, and who else is involved. Say the words "you are now the incident commander" and have it acknowledged, because an ambiguous handover leaves an incident with either two commanders or none.
The review has to change something
A review earns its cost only if it produces work that gets done. That means each action item has a named owner, a due date, and a place in the team's normal backlog rather than in the document — action items tracked only inside postmortems are not tracked. Prefer a small number of items that will genuinely land over a comprehensive list that will not.
Blameless is a technical discipline before it is a kindness. If the conclusion is that someone ran the wrong command, the review has stopped one step too early: why was that command available, why did it not require confirmation, why did nothing catch it for eleven minutes, and why did the runbook read as though it were safe. Write contributing factors in the plural, because a single root cause is almost always an artefact of stopping when you found something blameable. Be careful too with counterfactuals — "if only they had checked the dashboard" describes a world that did not happen and teaches nothing, whereas "the dashboard did not show the affected region" is a fact you can act on.
Record the times that let you compare incidents: when it started, when it was detected, when it was mitigated, when it was resolved. A pattern of long detection times is an alerting problem, and a pattern of long mitigation times is a tooling problem, and neither is visible from any single incident.
The quality of an incident response is decided by how quickly you were willing to act without understanding, and the quality of the review by how much of the system you were willing to blame instead of the person.
Likely follow-ups
- You are the only person online and the mitigation needs two pairs of hands. What do you do first?
- How do you decide between rolling back and rolling forward with a fix?
- A postmortem concludes that an engineer ran the wrong command. What is missing from that conclusion?
- What would make you page a human at 3am versus raise a ticket for the morning?
Related questions
- You are on call, a deploy went out twenty minutes ago and production is degraded. Talk me through what you do.hardAlso on incident-response and on-call7 min
- The line has been stopped for twenty minutes and everyone in the room believes it is your system. How do you run the next hour?hardAlso on incident-response and postmortem5 min
- Tell me about a decision you had to make in minutes, with information you knew was incomplete.mediumAlso on incident-response6 min
- You are mid-demo and the product fails in front of the customer. What do you do, and what do you do afterwards?mediumAlso on communication6 min
- What is the STAR method, and why do interviewers structure behavioural rounds around it?easyAlso on communication6 min
- Tell me about yourself — walk me through your resume.easyAlso on communication4 min
- In the room, the customer tells you a competitor has committed to something you cannot match. How do you respond?hardSame kind of round: scenario5 min
- The domain expert tells you one thing and the written procedure says another. How do you work out which one the system should follow?hardSame kind of round: scenario5 min