Tell me about a time you broke something in production. What happened?
Own the change that caused it, walk the timeline from detection to recovery in real minutes, separate your mistake from the system that let it through, and say which fix is still in place today. Interviewers score the honesty of the timeline before the cleverness of the fix.
What the interviewer is scoring
- Does the candidate name their own change as the cause without immediately diffusing it across the team
- Whether the timeline has real detail about how the problem was noticed and by whom
- That the impact is described in terms of users or money rather than "some errors in the logs"
- Whether the candidate separates the personal mistake from the missing guardrail that let it ship
- Does the remedial change still exist, and can the candidate say who owns it now
Answer
Why this question is asked instead of "tell me about a failure"
A generic failure question invites a curated answer, and most candidates arrive with one already polished. Asking specifically about breaking production narrows the ground so that evasion becomes visible. There is a change, a timestamp, a set of affected users and a recovery, and an interviewer who has run incidents knows what a real account of one sounds like. The question is a truthfulness check dressed as a technical one.
What is being assessed is a sequence rather than a virtue. Did you notice, or did somebody else have to tell you? Did you communicate while the cause was still unknown, which is when communication is hardest and most valuable? Did you stabilise first and diagnose second, or did you keep debugging while customers were still failing? And afterwards, did anything actually change, or did the team write a document and move on? Those four points, in that order, are what most interviewers are listening for, whether or not they say so.
Start with the change, not the symptom
The strongest openings are unglamorous and specific. Name what you did and when: a migration run against the wrong environment, a feature flag defaulted the wrong way, a config change that removed a timeout, a deployment that skipped a step because you were doing it manually at the end of a long day. Then say what broke and for whom, in the terms the business would use. "Checkout failed for about a fifth of sessions for nineteen minutes" tells the interviewer you know what your service is for. "There were a lot of 500s" tells them you watched a dashboard.
Then walk the clock. The detail that carries the most weight is how you found out, because it distinguishes a team with working instrumentation from a team that learns about outages from customers. If it was a customer, say so plainly and treat it as the second finding of the incident. That admission almost always improves the score, because the alternative is a story where alerts conveniently fired within a minute, and that story is the one interviewers most often doubt.
Language that keeps the ownership visible
Ownership evaporates in the grammar rather than in the facts. Candidates rarely deny responsibility outright; they let the sentence drift into the passive or into the plural, and the interviewer notices immediately.
| What candidates say | What it reads as | Say instead |
|---|---|---|
| "The migration was run against prod." | Nobody did it | "I ran the migration against production." |
| "We decided to roll back." | Somebody else decided | "I called the rollback about eight minutes in." |
| "There was a gap in the release process." | Pre-emptive deflection | "I skipped the checklist step, and the process let me." |
| "It got escalated and resolved." | You were a bystander | "I paged the on-call DBA and stayed on the call until we had restored." |
The last column is not braver, it is more useful, and it costs you nothing. Interviewers do not reduce scores for causing an incident; almost everyone in the room has caused one. They reduce scores for a narrator who is somehow absent from their own story.
Separating your mistake from the system's
This is where senior answers diverge from adequate ones, and where the balance is delicate. A candidate who only blames themselves — "I was careless, I have learned to be more careful" — has offered no engineering, because carefulness is not a control and will not survive the next tired Friday. A candidate who only blames the system sounds as though they are managing the interview.
The answer that lands does both, in that order. You made a specific mistake, and it reached users because a specific safeguard was missing or bypassed: no confirmation on destructive commands against production, no staged rollout, a flag with an unsafe default, no alert on the metric that actually mattered, a runbook nobody had rehearsed. Then say what you changed, and be honest about how far it got. "We added the guard for that one command and I have wanted to do the wider audit ever since, but it has not been prioritised" is a more credible sentence than a claim that the whole class of problem was eliminated, and it invites a conversation about influence rather than closing one down.
The recovery detail candidates skip
Very few answers mention what happened to the users. Whether anyone contacted the affected customers, whether data needed reconciling afterwards, whether a batch had to be replayed, whether support was given something to say while the cause was unknown — these are the parts of an incident that separate people who have carried real operational responsibility from people who have watched one from a chat channel. If you handled any of it, say so. It is the most underused evidence available in this answer.
Keep the emotional content short and factual. Saying that you felt sick and that it was the worst afternoon of that year is human and fine in one sentence. Two minutes of it shifts the interviewer's attention from your judgement to your resilience, and resilience is not what they set out to measure.
Interviewers are not scoring whether you broke production. They are scoring whether you can be trusted to say so early, in the first person, while it is still going wrong.
Likely follow-ups
- How did you find out, and would you have found out if a customer had not told you?
- Who did you inform first, and what did you say to them while you still did not know the cause?
- What did you consider doing that you decided against during the incident?
- What stops the same class of mistake today, and has it been tested since?
Related questions
- Tell me about a time you failed.mediumAlso on failure and accountability4 min
- You are on call, a deploy went out twenty minutes ago and production is degraded. Talk me through what you do.hardAlso on incident-response and postmortem7 min
- You get paged, the service is down, and nobody else is online yet. How do you run the incident?mediumAlso on incident-response and postmortem4 min
- The line has been stopped for twenty minutes and everyone in the room believes it is your system. How do you run the next hour?hardAlso on incident-response and postmortem5 min
- Tell me about a time you had far more work than you could deliver. How did you handle it?mediumAlso on behavioural5 min
- What is the piece of work you are proudest of, and why that one?mediumAlso on behavioural4 min
- There is a gap of just over a year on your CV. What were you doing?mediumAlso on behavioural4 min
- Tell me about a time you had to work closely with someone you found difficult.mediumAlso on behavioural5 min