You are on call, a deploy went out twenty minutes ago and production is degraded. Talk me through what you do.

The first two minutes are for scope, not for causes

Before deciding anything, establish how bad this is from a signal that describes users rather than machines. Error rate and latency at the edge, checkout or login success rate, queue age for an asynchronous path. "Degraded" is not actionable; "8% of authenticated requests are failing and p99 has gone from 300ms to 4s, starting at 14:12" is, and the timestamp is half the diagnosis because it either lines up with the deploy or it does not.

Two facts then decide the whole shape of the response. How many users are affected, which sets urgency and who needs telling. And what changed in the window before the inflection, because a deploy twenty minutes ago is the highest-prior hypothesis available and should be treated as the cause until something disproves it, not treated as one theory among many to be evaluated fairly.

Declare the incident at this point even if you suspect it will be brief. Declaring is cheap and creates the channel, the timeline and the roles; not declaring means the third person to notice starts their own uncoordinated investigation. If more than one person is now involved, separate the roles: someone commanding, who decides and does not type, and someone operating, who executes. The person who wrote the change is a valuable subject-matter expert and the worst possible decision-maker, because they are motivated to understand rather than to restore and they are the one who will be blamed by their own inner monologue.

Stabilise before you diagnose, and say that you are doing it

This is the discipline the question is really testing. With production degraded, the goal is to stop the harm, and understanding is not required to stop it. Reverting a deploy you do not yet understand is not sloppy, it is correct: it converts an open-ended investigation into a bounded one you can run against a healthy system, from a laptop, tomorrow, with sleep.

The mitigations available are broader than "roll back", and the cheapest one that works should win. A feature flag switched off is faster than a deployment and reversible per-feature, so four good changes in the same release are not withdrawn to undo one bad one. Aborting a canary or shifting traffic back is seconds where a full rolling redeploy is minutes. Scaling out buys time when the symptom is saturation rather than error. Shedding non-essential load or pausing a batch consumer keeps the core journey alive while you work. Choosing the narrowest action that restores the user-facing signal is a stronger answer than reaching for the biggest hammer.

The one thing worth thirty seconds before you destroy the evidence is capturing it, because a rollback or a restart erases the state that explains the failure. Take one artefact on the way out: logs from a failing instance, a thread or heap dump if it is a memory or lock symptom, the pod description, a screenshot of the dashboard with the timestamps visible. One artefact, thirty seconds, then revert.

The rollback decision, and when fix-forward is right

Rollback is the default because it is the option with the known outcome. Fix-forward is right in a small number of specific situations, and being able to name them is what separates a senior answer.

You fix forward when rollback is not actually available: a migration has contracted the schema so the previous binary can no longer read it, or the new version has been writing data in a format the old one will not understand, or an event with a new schema is already on a topic consumers have read. You fix forward when the fix is genuinely known, small and testable, and shipping it is faster than reverting, which is true for a bad configuration value and almost never true for a logic bug. And you fix forward when the previous version has a defect of its own that makes it no better.

Everything else rolls back. The way to keep that decision from consuming the incident is to time-box it aloud: "we try the config change for ten minutes, and at 14:45 we roll back regardless". Stating the deadline in the channel is what prevents the failure mode where a team spends ninety minutes almost fixing something while users are still failing, each successive five minutes feeling too small to justify abandoning the work already done.

Communication is a deliverable, not an interruption

Updates go out on a fixed cadence, and the cadence holds even when there is nothing new, because silence is read as chaos and generates a second incident made of people asking. Every update carries four things: what users are experiencing, what you currently believe, what you are doing about it, and when the next update comes.

14:28 — Investigating. Around 8% of logged-in users are seeing errors on checkout since 14:12. We have identified a deployment as the likely cause and are rolling it back now. Payments already completed are unaffected. Next update at 14:45 or sooner if this resolves.

Note what that does not do. It does not name the engineer, it does not speculate on root cause, it does not promise a resolution time you cannot control, and it does not use internal vocabulary. Anything customer-facing should be written in terms of what someone cannot do right now, with a clear statement of what is safe. Internally, keep one channel as the record, timestamp decisions as they are made rather than reconstructing them later, and make sure exactly one person owns talking to the business so the responders are not answering the same question five times.

The incident does not end when the code is reverted

The most common gap in an otherwise good answer is stopping here. Reverting the deploy restores the code path; it does not undo what the bad code did while it ran, and the second half of the incident is usually that repair.

Ask what accumulated during those twenty minutes. A queue backlog will now be consumed at a rate that may itself overload something downstream, so drain deliberately rather than by releasing the brake all at once. Failed jobs sitting in a dead-letter queue need replay, and replaying them against the reverted code is a decision rather than a formality. A cache may hold poisoned entries that survive the deploy and keep serving the bug. Rows written with a wrong value need a correcting query, written carefully and reviewed. And retrying clients may have created duplicate orders or duplicate notifications to partners, which is a customer-facing problem long after your dashboards are green.

Declare recovery on evidence: the user-facing signal back inside target for a sustained window rather than for one scrape interval, backlogs drained, no repair work outstanding. Then hand over properly if it is still the middle of the night, with a written state of the world rather than a verbal summary to someone half awake.

What the follow-up looks like

Write the postmortem within a few days while people still remember what they were thinking, and build the timeline from data — deploy timestamps, alert times, chat messages — rather than from memory, which reliably compresses and reorders.

The judgement that shows seniority is in the framing. Separate the trigger from the vulnerability: the deploy triggered it, but the reason a routine deploy could degrade production is the vulnerability, and that is the interesting half. Then ask the timing questions, because they generalise where the bug does not. Did an alert fire at 14:12, or did a customer tell you at 14:20? Was there a canary stage, and if so, why did it pass? Time to mitigate is a separate number from time to detect, and improving whichever dominated is worth more than fixing this specific bug.

Keep the actions few and owned, because three items with a named owner and a date get done while nineteen get admired. Prefer changes that remove a class of failure over changes that add a step to a runbook, and make at least one about detection rather than prevention, since the next incident will have a different cause and the same detection path. Blameless has to mean structurally blameless rather than politely blameless: the engineer who deployed acted on the information the system gave them, and if that information was insufficient, the system is the finding.

The order is stabilise, communicate, then diagnose — and the incident stays open until the data, the queues and the customers are repaired, not until the deploy is reverted.