AI in the Software Delivery Lifecycle: The Complete Interview Guide
How to answer the AI-in-delivery question that now appears in almost every engineering management, architecture and senior engineering loop: where assistance genuinely lands across the lifecycle, what speeds up and what does not, why percentage productivity claims collapse under questioning, how measurement fails, where the bottleneck moves, and what to say when your own experience is thin.
Masterclass·75 min read
What it is
Somewhere in the last two years, a question that used to be a curiosity at the end of an interview became a load-bearing part of the loop. An engineering manager, an architect or a senior engineer will be asked, in some form, what AI has changed about how their team builds software. It arrives phrased casually — "so, how is your team using AI?" — and it is graded like a system design round. The casualness is deliberate. It is a question with no correct position, only a correct quality of reasoning, and the easiest way to find out whether a candidate reasons or recites is to ask something that invites recitation.
Most candidates fail it in one of two symmetrical ways. The first is the evangelist, who reports a large percentage improvement, describes a transformed team, and cannot answer a single follow-up about how the number was arrived at. The second is the dismisser, who explains that it generates plausible nonsense, that they tried it and it was not useful, and that real engineering is about understanding systems. Both answers are cheap in exactly the same way: they let the candidate hold a position without doing the work of measuring anything. An interviewer who has run this question thirty times can sort candidates into evangelist, dismisser and the small third group within about ninety seconds, and only the third group is being evaluated on anything else.
The third group does something specific. They separate the parts of the delivery lifecycle where assistance genuinely removes work from the parts where it moves work somewhere else, they are precise about what they observed rather than what they believe, they can say what they measured and why the measurement is weak, and they can name a cost that the adoption created. That is the whole answer, and everything in this guide is an elaboration of it.
The subject has a shape, and it is worth having the shape in mind before the argument starts, because almost every strong answer to almost every question in this space is a claim about where in the lifecycle something changed.
Loading diagram…
Watch the two dotted edges rather than the solid path: generation volume pushes into review, and the verification effort that review discovers flows back into implementation, which is the mechanism by which a team gets faster at producing change and no faster at shipping it.
The three claims people conflate
Almost every muddled conversation about AI and productivity is a conflation of three different claims, held at three different levels of confidence, and stated as if they were one.
The first claim is about generation speed. A developer can produce a working first draft of a function, a test file, a migration script or a Terraform module faster than they could type it. This claim is close to uncontroversial and is directly observable by anyone who has used the tools for an afternoon. It is also the least interesting of the three, because typing was rarely the constraint on a delivery organisation.
The second claim is about delivery throughput. The team ships more change per unit time, or the same change sooner. This is a much stronger claim, it is the one an executive cares about, and it depends entirely on whether generation was on the critical path. If a change spends four hours being written and six days waiting — for review, for a shared test environment, for a release window, for a decision from another team — then halving the four hours moves the six days not at all. Most candidates who assert the second claim have only evidence for the first.
The third claim is about business outcome: the organisation delivers more value, or the same value with fewer people. This is the claim funding the tooling budget, and it is the hardest to support, because it requires that the additional throughput was spent on things that mattered, which is a prioritisation property rather than an engineering one. A team that ships twice as many features nobody uses has improved the second claim and worsened the third.
An interviewer asking "how much faster is your team" is usually asking about the second claim while the candidate answers the first. Naming the distinction out loud is one of the cheapest ways to signal that you have thought about this seriously, and it costs one sentence: "I can tell you what changed at the keyboard, and I can tell you what changed in our cycle time, and they are not the same number."
What the question is testing at each level
The question is asked of different roles for different reasons, and the same answer does not score equally everywhere.
Asked of a senior engineer, it is testing craft judgement: whether you know which tasks you delegate to a generator and which you refuse to, whether you read what you commit, and whether you can describe a case where the output was subtly wrong in a way that a test would not have caught. The interviewer wants to know whether your standard for "it works" changed.
Asked of a staff engineer or architect, it is testing systems reasoning: whether you can locate the constraint in a delivery system, whether you understand that relieving a non-constraint produces nothing, and whether you have thought about what large volumes of generated code do to a codebase's long-term comprehensibility. This is where the modular monolith against microservices trade-off style of reasoning gets applied to a delivery process instead of a runtime.
Asked of an engineering manager or director, it is testing three things simultaneously: whether you can measure something honestly, whether you can manage an adoption without either mandating it or ignoring it, and whether you can talk to your own leadership about it without either overpromising or looking like an obstacle. The last of those is the real test. Managers are increasingly under pressure to report an AI productivity number upwards, and the interviewer wants to know what you do when asked for one you cannot support.
Asked of a head of engineering or CTO candidate, it becomes a capital allocation and risk question: what you spend, what you forbid, what you tell the board, and what your position is on the data boundary.
The vocabulary interviewers listen for
There is a small set of distinctions that reliably separate people who have run this in an organisation from people who have read about it. None of them is jargon; each is a distinction that only matters once you have hit the problem.
Generation against verification. Draft against decision. Assistance against autonomy. Individual velocity against system throughput. Code written against change delivered. Accepted suggestion against retained code. The candidate who uses two or three of these unprompted has usually earned the benefit of the doubt on the rest of the answer, because these are the distinctions that only appear after somebody has tried to explain a disappointing result to a sceptical finance director.
The question has become standard because organisations made a large, fast, mostly unmeasured investment and now have to decide whether to make a larger one, and there is a shortage of people who can reason about it without a position to defend.
The vendors have a position. Their published figures are marketing artefacts, produced under conditions they controlled, usually measuring task completion time on bounded exercises rather than delivery outcomes on real systems, and typically reported without the denominator that would make them meaningful. This is not an accusation of dishonesty; it is a statement about what a vendor benchmark is for. An interviewer who hears a vendor figure repeated as evidence learns that the candidate does not distinguish between a marketing claim and a finding, and that inference is fatal in a role whose main output is other people's decisions.
The internal enthusiasts also have a position, and it is usually sincere. A developer who has just had a genuinely delightful afternoon with a tool will report a transformation, and they are reporting their experience accurately. The gap between that report and the delivery data is one of the most reliable findings in this whole area: people consistently feel faster, and the felt speed does not reliably show up in cycle time. There are good explanations for that gap which do not require anyone to be lying. Reduced cognitive friction feels like speed. Time spent waiting is not remembered as vividly as time spent typing. And the parts of the work that got slower — reading, checking, correcting — do not feel like work being added, they feel like normal engineering.
The sceptics have a position too, and it is not a safer one. An engineering leader who dismisses the whole area is making a bet about their organisation's cost base against a technology their competitors are using, and they are also telling their own senior engineers that a thing those engineers use daily is not worth discussing. Dismissal reads as incuriosity, and incuriosity is disqualifying at any level above senior.
So the interview question exists to find the person who has none of those three positions: who has looked, has a view about where it helped, has a view about where it did not, and can hold both without collapsing into a slogan.
What the organisation is actually worried about
Behind the question there are usually three anxieties, and it helps to know which one you are being asked about, because the strong answer differs.
The first anxiety is cost. Somebody has been told that engineering can be smaller or can absorb more work without growing, and they need to know whether that is true. The honest answer involves the distinction between the three claims above, and it involves saying that headcount reductions justified by generation speed are betting on the wrong claim.
The second anxiety is quality and liability. Somebody has read about generated code containing a licence-encumbered fragment, or a security flaw, or a subtle logic error that passed review because it looked idiomatic, and they need to know that the person they hire has thought about the control. This is where questions about ownership, review policy and data boundaries come from.
The third anxiety is capability decay. Somebody senior has noticed that a junior engineer produced a correct pull request and could not explain it, and they are worried about what their engineering organisation looks like in five years. This anxiety is rarely stated directly and is frequently the real content of a question phrased as "how do you onboard juniors now".
Recognising which anxiety is live in the room changes the emphasis of your answer, and asking is legitimate: "Are you asking me about the productivity case, or about the quality controls, or about what it does to how people learn? I have a view on all three but they go in different directions."
Where assistance genuinely lands in the lifecycle
The single most useful thing you can bring to this question is a stage-by-stage account, because it converts a slogan into an argument. What follows is a map of the delivery lifecycle with the honest position at each stage, and the honest position is almost never "faster" without qualification. It is "this part got cheaper and this other part got more expensive, and here is the net".
Deciding what to build, and getting a decision out of a stakeholder
Plausible-looking acceptance criteria that were never argued about by the people who care
Design exploration
Enumerating options, producing a straw-man design to react to, drafting an ADR
Choosing between options under real constraints
Anchoring on the first generated design because it arrived complete and confident
Implementation
First drafts, boilerplate, glue, unfamiliar-API scaffolding, one-off scripts
Understanding the existing system well enough to change it safely
Reading and correcting code you did not write, at volume
Test generation
Test scaffolding, table-driven cases, fixtures, coverage of obvious branches
Knowing which behaviours matter enough to be pinned
Tests that assert current behaviour rather than intended behaviour, and therefore ratify bugs
Code review
Style, obvious defects, missing null handling, first-pass summarisation of a large diff
Judging whether the change is the right change for this system
More diff arriving per reviewer per week
Documentation
Bringing stale docs into line, drafting runbooks, summarising a subsystem for a newcomer
Deciding what is worth documenting
Confident documentation of behaviour that was inferred rather than verified
Migration and modernisation
Mechanical translation, repetitive refactors, dependency upgrades across many modules
Deciding the target state and the cutover sequence
A large, uniform diff that no single person has read end to end
Incident triage
Log summarisation, hypothesis generation, correlating a change window with a symptom
Deciding to roll back, and having the authority to
Confident wrong hypotheses during the worst twenty minutes of the week
Customer support and internal enablement
Answering repeat questions from documented material, drafting responses
Handling the case that is not in the corpus
Answers that are wrong in the same tone as the answers that are right
The pattern down that table is the argument. In every row, the thing that gets cheaper is producing a candidate artefact, and the thing that does not move is deciding. The new cost, in every row, is some form of verification. If you can say that sentence in an interview and then support it with two concrete examples from your own experience, you have given a better answer than most of the field.
Requirements and refinement
This is where the returns are more real than people expect and less transformative than they hope. Drafting a story from a conversation, generating a list of edge cases against a stated behaviour, turning a rambling stakeholder note into something a team can refine — all of that genuinely removes drudgery, and the edge-case enumeration in particular is useful because it is a recall task and recall is exactly where a language model is strong. A refinement session where somebody has pre-generated twenty candidate edge cases and the team spends the hour arguing about which eight matter is a better hour than one spent trying to think of edge cases from cold.
What does not move is the part that was always the bottleneck: getting the person who knows what the business needs to make a decision and stand behind it. Refinement is slow because it is a negotiation between people with different information and different incentives, not because writing the story took a long time. A team that generates its acceptance criteria and skips the argument has automated the artefact and lost the point, which is the same failure mode as a backlog too big to refine: the volume of tickets was never the problem, the absence of a shared decision was.
There is a specific hazard worth naming, because interviewers who have lived through it will recognise it. Generated acceptance criteria read as authoritative. They are well-formed, complete-looking and impersonal, which is exactly the register that discourages challenge. A hand-written criterion that says "customer should probably see the old balance until midnight?" invites a conversation. A generated one that says "the system shall display the previous end-of-day balance until 00:00 UTC" invites agreement. The second is more likely to ship and less likely to be correct.
Design exploration
Design is a genuinely good use, provided you are clear that the output is a straw man rather than a proposal. Asking for four approaches to a problem, with their failure modes, produces a better starting list than most engineers produce alone, principally because it does not get bored or attached. Using it to draft the first version of a decision record, which you then rewrite with the real constraints, is faster than starting from an empty file — and the discipline that makes it safe is the same one described in what belongs in an ADR and when to write one: the value of the record is the rejected options and the constraints that forced the choice, and those are precisely the parts a generator cannot know.
The failure mode is anchoring, and it is stronger here than anywhere else in the lifecycle. A design that arrives complete, coherent and confidently argued suppresses divergent thinking in a way that a colleague's half-formed suggestion does not. Teams that use it well tend to have a rule about it: generate options, never a recommendation; or generate the recommendation and then require somebody to argue the opposite before it is adopted. A candidate who mentions an explicit countermeasure to anchoring is showing something that only comes from having watched a team adopt a bad design quickly.
The other thing that does not move is constraint discovery. The reason your design cannot use that queue is that the platform team is decommissioning it in Q3, and the reason you cannot split that table is a reporting job nobody owns. Those facts live in people, not in the repository, which is why inheriting a system nobody understands is still a matter of finding the three people who remember.
Implementation
This is where the visible gain is, and it is real. Boilerplate, glue code, adapters, one-off scripts, test fixtures, configuration, and the first ninety per cent of a function whose shape you already know — those are genuinely faster, and the gain is largest when working in an unfamiliar language or against an unfamiliar API, where the alternative was forty minutes of documentation reading.
Two qualifications matter, and stating them is what distinguishes a considered answer.
The first is that the gain is concentrated in greenfield and well-trodden work and decays sharply in legacy and idiosyncratic work. A generator works from patterns; a fifteen-year-old billing system with three overlapping abstractions for the same concept and a naming convention nobody remembers the reason for is, by construction, out of distribution. The teams that report the largest gains are frequently the teams doing the newest work, which is a selection effect rather than a finding, and an interviewer who has noticed this will probe for it.
The second is that the acceptance-to-retention gap is real and rarely measured. Code suggested is not code accepted, code accepted is not code committed, and code committed is not code that survives the next month. Any figure quoted at the top of that funnel — suggestions accepted, percentage of lines generated — describes the top of the funnel and nothing else. If you are going to quote anything, quote something from the bottom.
Test generation
Test generation is the stage with the widest gap between how good it looks and how good it is, and it is worth being specific because interviewers use it as a depth probe.
What works well: scaffolding, fixture construction, parameterised cases across an enumerable input space, and filling in the boring branches of a function whose intended behaviour is already pinned by a few hand-written tests. Coverage of the obvious rises quickly and cheaply, and coverage of the obvious does have value.
What fails, and fails quietly, is that a generator writing tests against existing code writes tests that assert what the code does, not what it should do. Point it at a function with an off-by-one error and it will produce a beautifully structured test suite that permanently ratifies the off-by-one. The suite then makes the bug harder to fix, because fixing it breaks tests, and breaking tests is a signal engineers are trained to treat as evidence they are wrong. This is a genuine, mechanical, easily explained failure mode, and being able to state it is one of the highest-value sentences available in this whole interview area. It also has a clean countermeasure: generated tests are acceptable for regression pinning and unacceptable as the specification, so the tests that encode intended behaviour are written by a person, from the requirement, ideally before the code exists.
The related point is that coverage as a number becomes even less meaningful than it was. It was always a weak proxy; when the marginal cost of adding a test approaches zero, a coverage target stops being a constraint on anything and becomes a number the team can produce on demand.
Code review
Review is the stage where the interesting thing happens, and it deserves its own section, which it gets below. In summary: automated review is genuinely useful for the mechanical layer — style, obvious omissions, missing error handling, a first-pass summary of a large diff so the human reviewer knows where to look — and it is not useful for the layer that matters, which is whether this change is the right change for this system given everything not written down.
Documentation
The economics here are the most favourable in the lifecycle, for an unglamorous reason: documentation was previously not written at all, so the comparison is against nothing rather than against a human-quality alternative. A generated runbook that is eighty per cent right is a large improvement on an absent runbook. A summary of an unfamiliar subsystem for a new joiner is useful even when imperfect, because the new joiner is going to verify it by reading the code anyway.
The hazard is that documentation is trusted differently from code. Nobody runs documentation, so nothing fails when it is wrong; it simply misleads somebody at three in the morning. Generated documentation that describes inferred behaviour with the same confidence as observed behaviour is worse than no documentation for exactly the class of reader who most needs it. The teams that handle this well mark provenance: generated and unreviewed, generated and verified, or written by a person who owns it. That practice also addresses the older problem in keeping architecture documentation from going stale, because the cost of regeneration is now low enough that staleness is a process choice rather than an inevitability.
Migration and modernisation
Large mechanical migrations are among the clearest wins available, and they are also where the review problem is most acute. Converting a thousand test files from one framework to another, updating a deprecated API call across two hundred modules, or translating a well-understood service from one language to another are tasks where the work is repetitive, the correctness criterion is mostly local, and the alternative was a quarter of somebody's life.
What has not changed at all is the hard part of a migration, which was never the transformation. It was deciding the target state, sequencing the cutover, running two systems in parallel, and knowing what rollback means once data has been written on both sides — the substance of choosing the cutover day and what rollback means and of replacing a system that cannot be switched off. A team that generates the migration in a week and then spends six months on cutover has learnt something important about where its time actually went.
The reviewing problem is genuine and worth naming: a fifteen-thousand-line uniform diff is not reviewable by reading it. It has to be verified by a different instrument — a characterisation test suite captured before the change, a shadow run comparing outputs, a staged rollout — and a candidate who says "we changed how we reviewed it because it could not be read" is describing exactly the adaptation the situation demands.
Incident triage
Assistance during an incident is genuinely valuable in a narrow band and genuinely dangerous outside it. Summarising a large volume of logs, correlating a symptom window with a deployment window, and producing a list of candidate hypotheses are all tasks where speed matters and where being partially right is useful. Drafting the customer-facing status update while the responders keep working is a small, real saving at a moment when writing is expensive.
The danger is that a confidently phrased wrong hypothesis during the first twenty minutes of an incident costs more than no hypothesis, because incident response is a search process and a false lead consumes the scarce resource, which is the attention of the two people who understand the system. The teams that use it well tend to treat generated hypotheses as items on a list to be checked in cost order, not as a diagnosis, and they keep the decision to roll back with a named human — which it had to be anyway, since rollback is a judgement about blast radius and customer impact rather than a technical conclusion.
Support and enablement
Internal support — the question channel where developers ask the platform team the same eight questions — is a straightforwardly good application, provided the corpus is good. The failure is uniform tone: the retrieval-backed answer that is wrong sounds exactly like the one that is right, and the reader has no way to distinguish them. Teams that get this right make the sourcing visible, so a reader can see which document the answer came from, and treat "no confident source" as an answer rather than a prompt to improvise.
What gets faster, and why the total often does not
The most common analytical error in this whole discussion is treating a delivery organisation as if its output were proportional to how fast its engineers write code. It is not, and it has not been for a long time, which is why the discipline of looking at where a unit of work actually spends its elapsed time is the single most useful thing to bring into this conversation.
Take a typical change in a mid-sized organisation and account for its wall-clock life from first commit to production. Some of it is a person writing code. Rather more of it is the change sitting in a queue: waiting for review, waiting for a reviewer who has context, waiting for a shared environment, waiting for a dependent team, waiting for a release window, waiting for a decision. The proportions vary enormously between organisations, which is exactly the point — you cannot know the effect of speeding up authoring until you know what fraction of elapsed time authoring was.
Where elapsed time goes
Effect of faster generation
Why
Understanding the problem and the existing code
Slight, sometimes negative
Comprehension is the work, and reading unfamiliar generated code adds to it
Writing the first draft
Large
This is the part that is genuinely automated
Self-checking and local iteration
Neutral to worse
More candidate code to check per unit of thinking
Waiting for review
Worse if volume rises
Reviewer capacity is fixed and demand went up
Rework after review
Depends entirely on quality
Faster to produce a revision, but more revisions if the first draft was subtly wrong
Waiting for environments, dependencies, other teams
None
Untouched by anything happening at the keyboard
Release, verification, rollout
None, unless deployment itself is automated separately
A different constraint with a different fix
The honest summary is that generation speed improves one row of that table decisively and several others not at all, and that the effect on the total depends on the shape of your particular organisation's table. In a team where a change waits nine days for a reviewer, faster authoring produces almost nothing and may produce a longer queue. In a team with fast review, good automated verification and continuous deployment, it produces a real gain, because there is nothing downstream absorbing it.
That is a genuinely interesting thing to say in an interview, and it also generalises. This is Amdahl's law applied to a delivery process: the improvement available from optimising a stage is bounded by that stage's share of the total. Candidates who make this argument in one sentence, and then say what their own organisation's constraint was, are demonstrating something that cannot be faked.
There is a second-order effect worth mentioning because senior interviewers listen for it. Relieving a non-constraint does not merely fail to help; it can hurt, by increasing the arrival rate at the constraint. A team that produces change faster and reviews it at the same rate has a growing queue, and a growing queue means larger batches, longer feedback loops, more merge conflicts, more context lost between authoring and review, and more rework. That is the mechanism by which a team can adopt a productivity tool and get slower, and it is the failure that a well-run measurement programme would catch and a badly run one would not.
Why percentage productivity claims collapse
Someone will eventually ask you, or ask your team, for a number. Understanding precisely why the numbers on offer do not survive scrutiny is what lets you decline to produce one without sounding evasive.
The first problem is the counterfactual. A productivity claim is a comparison against what would have happened otherwise, and that comparison does not exist. You did not run the same quarter twice. Everything else changed too: the team composition, the codebase, the nature of the work, the amount of production firefighting, the maturity of the product. Attributing a delivery change to one variable in an uncontrolled system with a dozen moving parts is the sort of claim that a numerate interviewer will pick apart in two follow-up questions.
The second problem is selection. Developers use assistance where they expect it to help and do not use it where they do not, so any comparison between assisted and unassisted work is comparing different work. Tasks that get delegated to a generator are systematically the more routine ones. Measuring that those tasks completed quickly demonstrates that routine tasks are routine.
The third problem is the unit. Faster at what? Completing a bounded exercise in a laboratory tells you about bounded exercises. Time to first draft tells you about first drafts. Neither is time to production, and time to production is the only unit anyone outside engineering is buying.
The fourth is self-report. Surveys of perceived productivity are useful data about morale, friction and tool satisfaction, and they are close to worthless as evidence about throughput, because perceived effort and elapsed time diverge systematically. This is not a criticism of the developers answering; it is a known property of how people estimate their own time. Reporting a survey result as a productivity figure without labelling it as perception is a straightforward error, and interviewers catch it.
The fifth is survivorship in the reporting. Teams with disappointing results do not publish. Vendors publish results they commissioned. Conference talks are given by people whose adoption went well. The visible evidence base is filtered before you see it, and the filter is not random.
The claim you will hear
What it usually rests on
The follow-up that dissolves it
"We are thirty per cent more productive"
Perceived time saved, self-reported
Thirty per cent of what, measured between which two points, against which baseline
"Forty per cent of our code is AI-generated"
Tool telemetry on accepted suggestions
How much of it survived a month, and what does the share of authorship have to do with output
"Our velocity went up two points"
Story points per sprint
Who estimated the points, and were they estimated after the team knew it was assisted work
"Pull requests per developer doubled"
Version-control counts
Did the change per pull request halve, and did cycle time move
"We shipped the migration in six weeks instead of six months"
A single project comparison
What was the six-month figure — an estimate, or a comparable delivered project
"Onboarding is much faster now"
New joiner sentiment
Time to first independent production change, before and after, same team
None of this means you have to say "we do not know". It means you say what you observed and how confident you are in it, which is a different and much stronger position. The formulation that works is a bounded, sourced, honestly-hedged claim: something you measured, between two defined points, with the confounders named out loud.
What to say when asked "how much faster is your team"
Here are phrasings that hold up under questioning. They are worth having close to word-perfect, because this exchange happens under mild pressure and the difference between a strong and a weak answer is largely one of composure.
"I do not have a productivity percentage I would defend, and I would be sceptical of anyone who gave you one. What I can tell you is what we measured. Our median pull request size fell, our review turnaround went from about two days to about one after we changed how we staffed review, and our change failure rate did not move. I would attribute the review change mostly to a policy change rather than the tooling."
"Two things moved that I am confident about. Our engineers spend noticeably less time on boilerplate, and our test coverage on new modules is higher because writing tests got cheap. What did not move was our lead time, and the reason is that our lead time is dominated by waiting for a shared integration environment, which no amount of faster typing fixes."
"Honestly, the clearest effect was not speed, it was that people take on tasks they used to avoid. Someone will now do the tedious dependency upgrade because it is a forty-minute job instead of a two-day one. I cannot put a number on it and I think it is genuinely valuable."
"I would want to answer that in two parts, because the number that gets quoted is usually generation speed and the number you care about is delivery. Can I tell you what we saw at each?"
Notice what those have in common. Each names a specific observable, each names a limit on the claim, and each declines the invitation to produce a percentage without sounding like a refusal. That last property is what makes them work in a room: the candidate has more to say, not less.
Measurement: what to use and what to refuse
If you are an engineering manager, this is the section of the interview where you win or lose the round, because measurement is the part of the topic that is genuinely your job.
The metrics that are worse than useless
Some proposed measures are not merely weak; they actively cause harm by changing behaviour in the wrong direction, and being able to say why is a strong signal.
Lines of code was a discredited measure before generation existed, and it is now actively inverted. Assistance makes it trivially easy to produce volume, and the highest-value engineering output — the change that deletes a subsystem, the design decision that avoids a project — has always registered as negative or zero. Measuring lines in an environment where lines are nearly free is measuring the tool's verbosity setting.
Pull request counts fail for the same reason with an extra twist. Splitting work into more pull requests is a good practice and doubles the metric without changing output, so the metric rewards a behaviour change that is orthogonal to value. Worse, it is trivially gameable and everybody knows it, which means the moment it is used for evaluation the number stops carrying information.
Suggestion acceptance rate is a vendor telemetry metric measuring engagement with a product, not output from a team. It answers the question "is the tool being used", which is a legitimate adoption question and an illegitimate productivity one.
Percentage of code written by AI is the most seductive of the bad metrics because it sounds like a measure of leverage. It is a measure of authorship share. A team where every line is generated and nothing ships is at one hundred per cent.
Story points per sprint deserves particular care, because it is the metric most likely to be waved at you by someone senior. Points are a relative sizing convention agreed within a team for forecasting purposes, they are not a unit of output, and they inflate under exactly the conditions where somebody is looking for a productivity improvement. The argument is set out fully in why story points and what if estimates are always wrong, and the specific hazard here is that if the team knows the number is being read as an AI productivity result, the number will move regardless of whether anything else does.
Proposed metric
What it really measures
How it gets gamed, usually without anyone deciding to
Lines of code
Verbosity, and the tool's default style
Accept the longer suggestion; stop deleting things
Pull requests per developer
Batching convention
Split the same work into more, smaller changes
Commits per day
Commit hygiene habits
Commit more often; nothing else changes
Suggestion acceptance rate
Engagement with the tool
Accept and then immediately edit; the acceptance still counts
Percentage of code AI-authored
Authorship share of whatever shipped
Generate the boilerplate you would have generated anyway
Story points per sprint
The team's current sizing convention
Sizes drift upward once the number is being watched
Test coverage
How much code a test executed
Generate assertion-light tests over the easy branches
Time saved, self-reported
Perceived friction, which is real but is not time
Nothing deliberate; perception simply diverges from elapsed time
What DORA-style metrics can and cannot do
The four delivery measures — lead time for changes, deployment frequency, change failure rate, and time to restore service — are the best instruments available here, and they are still not attribution instruments. Being precise about that distinction is the single most credible thing an engineering manager can say on this topic.
What they can do is tell you whether your delivery system got better or worse, in outcome terms that survive contact with a non-engineering audience. They are outcome measures rather than activity measures, which means they cannot be improved by working harder in the wrong place, and that property is exactly why they resist the gaming that destroys the activity metrics. If your lead time fell and your change failure rate held, something good happened to your delivery system. That is a genuine, useful, defensible statement.
What they cannot do is tell you why. They are a property of the whole system, they respond to everything, and in the period you adopted assistance you also changed three other things. A candidate who presents a DORA improvement as proof of an AI effect has made precisely the attribution error the metrics are least suited to supporting. The correct framing is that the four measures are a guardrail rather than a scoreboard: you adopt a change, you watch that the outcome measures do not degrade, and you attribute cautiously if at all. The broader discipline is covered in using DORA metrics without them becoming a stick, and the failure mode named there — the moment a delivery measure becomes a target applied to individuals it stops measuring delivery — applies with extra force when the measure is being used to justify an investment somebody has already made.
There is also a specific asymmetry that senior interviewers appreciate. Change failure rate and time to restore are the two that catch the downside of this particular adoption, and they are the two people forget to look at, because the story everyone is telling is about speed. If generated code carries subtly more defect risk, or if reviewers are approving more diff per hour than they can genuinely absorb, the signal shows up as escaped defects rather than as slower delivery. Watching only the speed pair is watching only the half of the instrument that can produce the answer you want.
What is worth measuring instead
A measurement programme that would satisfy a sceptical interviewer has four properties.
It measures flow rather than activity: cycle time distribution rather than counts, and the distribution rather than the mean, because the interesting change is usually in the tail. A team whose median cycle time is flat and whose ninety-fifth percentile halved has genuinely improved and a mean would have hidden it. The reasoning behind preferring flow measures is in which delivery metrics are worth tracking, and it applies unchanged here.
It measures where time is spent waiting, because that is where the answer to "did this help" actually lives. If you can break cycle time into authoring, waiting for review, in review, waiting for environment and waiting for release, you can say something meaningful about whether relieving authoring did anything. Most organisations cannot break it down at all, which is itself the finding.
It pairs a speed measure with a quality measure, always. Any claim about going faster that is not accompanied by a defect or failure measure is an incomplete claim, and the pairing is what makes it credible rather than promotional.
It is honest about the confounders in the write-up. Naming the three other things that changed in the same period is not a weakness in the report; it is the thing that makes the rest of the report believable to a reader who has seen a lot of reports.
And a fifth property, which is less about measurement than about integrity: it is agreed before the adoption, not selected afterwards from whatever moved. Choosing your metric after seeing the data is how an organisation produces a number it believes and cannot defend.
The review bottleneck moves rather than disappearing
This is the single most important structural insight in the topic, and a candidate who arrives with it already formed is immediately in the top group.
Consider what happens to the flow of work when generation gets cheap and nothing else changes. More candidate change is produced per developer-day. All of it still has to be understood by a second person before it enters the system. That second person's capacity is unchanged, and their throughput is bounded by reading comprehension, which no tool has yet made faster in the way that writing has become faster.
Loading diagram…
The edge to watch is the one from the queue back into reviewer attention: the queue does not just get longer, it changes what review is like, because a reviewer facing a backlog reviews differently from one who is not.
That last point is the substance. Reviewer behaviour under load is well understood by anybody who has managed a team through a busy quarter. Under a growing queue, reviews get shallower, approvals get faster, comments become stylistic rather than architectural, and the reviewer starts trusting the author's judgement in place of exercising their own. None of this is announced. The pull requests still get approved, the metrics still look fine, and the quality of the gate has silently degraded. This is the mechanism by which faster generation produces more escaped defects without anybody making a decision to accept more risk.
There is a further wrinkle specific to generated code, and it is the one worth raising because it is not obvious. Reviewing code written by a colleague is helped by a model of that colleague: you know what they habitually get wrong, you know when they are outside their comfort zone, you can read hesitation in a design. Generated code carries none of those cues. It is uniformly confident, uniformly idiomatic, and uniformly plausible, which strips the reviewer of the heuristics they had been using to allocate their attention. A reviewer who has always skimmed the boilerplate and concentrated on the parts where the author sounded unsure has lost their signal, and the failure looks like an experienced reviewer approving something that a less experienced one would have questioned.
The countermeasures teams actually use are worth being able to list, because "how would you handle that" is the obvious follow-up.
Keeping change small is the first and most effective, and it is now more important rather than less, because the marginal cost of producing a large change fell and the cost of reviewing one did not. A team that enforces a size limit on pull requests is not being fussy; it is defending the only stage that cannot be scaled by tooling.
Requiring the author to have read and understood everything they submit, and to be able to explain any line, is the second. This sounds obvious and is routinely not true in practice. Making it an explicit, stated norm — you own what you submit whether you typed it or not — changes behaviour more than people expect.
Shifting review effort towards the parts machines cannot check is the third. If a reviewer's time is finite and an automated pass genuinely handles style, formatting, obvious null handling and common error patterns, then the human should be reading for the things that require system knowledge: is this the right place for this logic, does it duplicate something, does it break an invariant that is not encoded anywhere, does it create a coupling we will regret.
Strengthening the automated gates around review is the fourth, and it is the one that scales. Property-based tests, contract tests at service boundaries, static analysis, dependency and licence scanning, and staged rollouts all catch classes of error at machine speed. The general point — that the reliable way to increase throughput through a human gate is to reduce what must pass through it — is the same one that shows up in coordinating dependencies across multiple teams: you do not fix a queue by asking the server to work faster.
Quality, defect escape and the verification asymmetry
The underlying economics are simple enough to state in one line, and stating it is a strong move in an interview: the cost of producing a candidate solution has fallen sharply, and the cost of establishing that a candidate solution is correct has not fallen at all.
That asymmetry has consequences that compound. When generating a solution was expensive, the expense acted as a filter — people thought before they wrote, they wrote less, and what they wrote carried the reasoning that produced it. When generation is nearly free, the filter is gone, and the only remaining filter is verification. Verification is now the whole quality system, and it is under more load than it was designed for.
What changed
Direction
Consequence for the delivery system
Cost of producing a candidate change
Down sharply
More candidates enter the system per unit time
Cost of establishing correctness
Flat, or slightly up
Verification becomes the binding constraint
Cost of producing a plausible wrong answer
Down sharply
Plausibility is decoupled from correctness in a way reviewers are not trained for
Cost of understanding an unfamiliar change
Up
The author may not hold the model of the code that the reviewer expects them to have
Cost of producing tests
Down sharply
Test volume rises, test meaning does not automatically follow
Cost of deleting code
Unchanged
Codebases grow faster than they shrink, and comprehensibility declines
The most under-discussed row is the last. Assistance is excellent at adding and indifferent at removing, because removal requires knowing what depends on a thing across a whole system and across time, which is precisely the knowledge that is not in the repository. Over a couple of years, a team that generates readily and deletes at the old rate ends up with a larger codebase per unit of delivered function, and codebase size is one of the better predictors of how hard the next change will be. That is a slow, invisible cost, it does not show up in any quarterly metric, and mentioning it marks a candidate as someone who thinks in years.
The defect profile also shifts in character rather than simply in volume, and this is a point worth making carefully because it is easy to overstate. Generated code tends to be syntactically clean and idiomatic, which means the classes of error it produces are not the classes reviewers are best at catching. The errors are less often typos and more often: a plausible but wrong assumption about a boundary condition; correct-looking handling of an error case that cannot occur alongside missing handling of one that can; an approach that works for the example and does not scale to production volumes; and, most insidiously, code that is subtly inconsistent with a convention or invariant that holds everywhere else in the codebase but is not written down anywhere. That last one is the signature failure. It is invisible in a diff, because a diff shows you the change and not the seventeen other places that assumed something different.
The junior-engineer problem
The question about what this does to people early in their careers is being asked more often, it is asked with real concern rather than as a test, and a considered answer is disproportionately valuable because most candidates have not thought about it.
The mechanism of concern is straightforward. Skill in software engineering is formed largely by struggle: by being stuck, forming a wrong hypothesis, testing it, being wrong again, and eventually building an accurate mental model of a system through the accumulated residue of those failures. The struggle is not incidental to the learning, it is the substance of it. A tool that removes the struggle removes the mechanism, and it removes it precisely at the stage where the mechanism is doing its most important work.
The observable symptom, which several people will tell you they have seen and which you should be careful to report as observation rather than as a study, is a junior engineer producing correct work they cannot explain and then being unable to debug it when it fails. The pull request is fine. The follow-up question is not. And because the pull request is fine, the gap does not surface through the normal channels — it surfaces during an incident, or during a design conversation eighteen months later when a promotion is being discussed and the person turns out not to have the depth their output implied.
There is a second, subtler mechanism: calibration. Part of what an engineer develops over their first years is a sense of when something is wrong. That sense is built from repeatedly seeing the difference between code that works and code that looks like it works. An engineer whose exposure has been mostly to plausible, confident, idiomatic output has fewer of those discriminations available. They cannot easily tell a good suggestion from a bad one, which is exactly the judgement the tool requires of them.
None of this argues for withholding the tools, and a candidate who proposes banning them for juniors is giving an answer that is both impractical and slightly ridiculous, since the tools are what the industry uses. What a strong answer proposes instead is deliberate practice design.
The measures that people actually use are worth being able to name. Requiring an explanation as part of review, so that "walk me through why this handles the retry case" is a routine question rather than an accusation, catches the gap early and cheaply. Reserving certain categories of work — debugging a production issue, tracing a request through the system, reading an unfamiliar subsystem and reporting on it — as explicitly unassisted, because those are comprehension exercises and the comprehension is the deliverable. Pairing on the hard problems specifically, since pairing preserves the struggle while making it survivable. And changing what is assessed in progression conversations, from output to explanation depth, which is the change that makes the rest of it stick.
That last point connects to how growth is evaluated generally. The criteria that get an engineer to senior — scope, judgement, the ability to take an ambiguous problem and return a decision — were never about typing speed, and they are the ones now doing more of the work in a promotion case. The framing in how do you get a mid-level engineer promoted to senior holds, but the evidence you gather has to shift: output volume was always a weak signal and is now a nearly meaningless one, so the evidence has to come from decisions the person made and problems they diagnosed. The same shift affects the other direction. Diagnosing genuine underperformance is harder when a struggling engineer's output looks superficially fine, and a manager who has not adjusted for that will find the situation described in an engineer on your team is underperforming surfacing much later than it used to, when it is more expensive to address.
Hiring feels the same pressure from the other end. A take-home exercise is now a test of prompt fluency unless it is designed otherwise, and a coding round that forbids assistance is testing a condition the job does not contain. The sensible reaction is not to police it but to move the assessment towards what cannot be delegated: reading unfamiliar code and explaining it, debugging something broken, defending a design choice under questioning, and reviewing a change and finding what is wrong with it. That is a redesign of the loop rather than a policy patch, and it interacts with everything in design an interview loop for a senior engineer.
Ownership and accountability for generated code
This question is asked in almost every architect and engineering management loop, usually phrased as "who owns AI-generated code", and it has a correct answer that most candidates arrive at slowly and should arrive at immediately.
The person who submits the change owns it, completely, exactly as if they had typed every character. There is no separate category of code with diminished ownership, because a category of code that nobody is accountable for is a category of code that nobody maintains, debugs at three in the morning, or takes responsibility for when it fails. Making this explicit is cheap and it forecloses an entire class of future argument.
What follows from that principle is a set of concrete obligations, and the strong answer states them rather than leaving them implied. You have read every line you submit. You can explain why it works. You would have been prepared to write it. If you would not have written it that way, you either change it or you can say why the generated approach is better. And when it breaks in production, "the tool wrote it" is not an available sentence.
There is a practical question underneath about provenance tracking. Some organisations want commit-level marking of which code was generated. The argument for is auditability, and in regulated environments it is sometimes a requirement rather than a preference. The arguments against are that it is unenforceable at the granularity people imagine, that the boundary is genuinely fuzzy once a developer has edited generated code and then regenerated part of it, and that it risks establishing the two-tier ownership the principle above exists to prevent. A candidate who can lay out both sides and then say what they would do in a specific regulatory context is answering better than one who has a policy.
There is also a liability layer that architects at senior levels are expected to have thought about, even if they are not lawyers. Whether generated output is copyrightable, who bears responsibility when a generated fragment reproduces licensed code, and how a customer contract's warranties interact with it are live questions with jurisdiction-dependent answers and no settled position. The correct interview behaviour here is precisely the same as in any area where the honest answer is unsettled: name the question, name who owns it in your organisation — legal, and not you — and describe the engineering control you put in place while the question remains open, which is usually licence scanning in the pipeline and a policy about what may be pasted where.
Security, licensing and the data boundary
For a large fraction of organisations, the binding constraint on adoption is not usefulness or cost. It is what may be sent where, and the candidate who understands that is answering a different and better question than the one about productivity.
Constraint
What it forbids in practice
What it drives teams towards
Customer data may not leave the tenancy
Pasting production data, real payloads or database contents into an external service
Synthetic fixtures, redaction in tooling, self-hosted or tenancy-bound models
Source code may not leave the tenancy
Any hosted assistant that transmits repository context off-premise
Enterprise agreements with no-training and no-retention terms, or self-hosted models
Regulated or classified environments
Egress from the development network at all
Air-gapped deployment, with a permanent capability lag against hosted offerings
Data residency requirements
Processing in a region outside the permitted set
Region-pinned deployments; often eliminates the preferred vendor
Secrets and credentials
Any prompt containing a key, token or connection string
Pre-flight scanning, and an incident process for when it happens anyway
Licence contamination risk
Accepting output that reproduces a copyleft-licensed fragment verbatim
Licence scanning in the pipeline, provenance indemnities in the contract
Third-party contractual confidentiality
Sending a client's code or specification to a subprocessor they have not approved
Per-client tooling policies, which are painful and are sometimes the only option
Audit and evidence obligations
Being unable to say what influenced a change in a regulated system
Logging, retention of prompts and outputs, change-record discipline
Three things are worth saying about this table in an interview.
The first is that these constraints are usually not negotiable by engineering, and the mistake candidates make is treating them as friction to be argued away. A data protection officer or a client's security team enforcing a residency requirement is doing their job, and the productive response is to find the deployment shape that satisfies it, not to build a business case for an exception. The parallel with any other compliance conversation is exact: you work with the reviewer, not around them.
The second is that the constraint set differs per client in a services organisation, and this is genuinely operationally hard. A consultancy with fifteen clients may have fifteen policies, and the practical answer is per-engagement tooling configuration, an onboarding checklist that states what is permitted, and a default of "assume not permitted until confirmed". A candidate who has lived this will mention the default, because the incidents all come from people assuming the general policy applied to a specific client.
The third is that the secrets problem is not hypothetical and is worth having a position on. Developers paste things. A configuration file containing a connection string will end up in a prompt, and the control has to assume it rather than forbid it, which means pre-flight scanning where you can put it and a rotation process for when you cannot. Treating this as a training problem rather than a control problem is the answer that tells an interviewer you have not run it.
Adoption is a change-management problem
The last substantive theme, and the one that most distinguishes an engineering management answer from an engineering one, is that the difficulty of adoption is almost never technical. Procuring licences takes a week. Getting an organisation to change how it works takes a year, and the failure modes are the ordinary failure modes of any change programme.
The mandate is the most common error. Requiring usage produces compliance behaviour: people use it where it does not help so that the telemetry looks right, the number goes up, nothing improves, and the credibility of the next initiative is spent. This is the same dynamic that sinks the transformations described in why do agile transformations fail, and for the same reason. A practice imposed without the reasoning behind it is performed rather than adopted, and performance is indistinguishable from adoption in any metric you can collect centrally.
The opposite error is pure laissez-faire, which produces wildly uneven practice, no shared norms about review or data boundaries, a handful of people quietly doing something risky, and no organisational learning at all. Everyone discovers the same pitfalls independently.
What works, as far as anyone can tell, looks like every other successful practice change. A small number of volunteers try it on real work. The organisation supports them with time rather than exhortation. They report honestly, including the parts that did not work, and the honesty is what makes the report credible to the sceptics. Norms get written down — what review looks like, what may be sent where, who owns the output — before scale rather than after the first incident. And the norms are owned by engineers rather than issued by management, which is the difference between a standard and an instruction.
There is a specific management skill in handling the two populations at the extremes. The enthusiast who wants to rebuild a working service with a new toolchain is expressing a real energy and pointing it at the wrong target, and the conversation is the one in the team wants to rewrite a service that works: the question is not whether the new way is better but whether the rewrite is the best available use of the quarter. The refuser is more delicate, because refusal is sometimes principled and sometimes fear, and treating a competent senior engineer's scepticism as a compliance problem is how you lose them. The move that works is to ask them to define the conditions under which they would use it, which converts a position into a testable proposition and usually surfaces the real objection, which is frequently about review load or code ownership rather than about the tool.
Two organisational constraints are worth naming because they are the ones that determine whether any of this is affordable.
The first is that adoption consumes capacity before it returns any. Learning, experimenting, writing norms, revising review practice and dealing with the first bad outcome all cost engineering time in the quarter when the benefits do not yet exist. A manager who has not budgeted for that is running an unfunded initiative, and the negotiation is the same one as in how much capacity goes to roadmap versus keeping the lights on: if it is not allocated, it comes out of somewhere unplanned and someone will be surprised.
The second is that the pitch upwards has to be honest to survive. A leader who funds tooling on a promised productivity percentage has created an obligation to produce that percentage, and when the delivery data refuses to cooperate, the choice is between a bad number and a bad conversation. Framing the investment the way you would frame any other platform investment — expected effect, how you will know, what you will stop doing if it does not work — avoids the trap entirely, and it is the framing set out in make the case for headcount or platform investment.
When leadership has already decided
A scenario increasingly used in interviews is the one where the decision has been made above you: headcount has been reduced or frozen on the assumption of an AI productivity gain, and you are being asked to deliver against it. The scenario is testing whether you can disagree usefully.
The answer that fails in both directions is either quiet compliance, which means committing to something you believe is undeliverable, or open opposition, which makes you a problem rather than a resource. The behaviour that scores is the same as in any commitment you do not believe: make the disagreement specific and written, state what you can commit to and on what basis, name what you would need for the larger commitment, and then execute the decision that is taken. That is exactly the structure in an executive wants a firm date for unscoped work and in three commitments and only two will land: you do not win by refusing, you win by making the trade-off visible early enough that somebody can choose.
Answering honestly when your own experience is thin
A large number of candidates are asked this question having used the tools personally and never run an organisational adoption. This is an entirely normal position and it is not the disadvantage people fear, provided it is handled with precision rather than with invention.
The rule that matters: do not manufacture experience. Interviewers ask this question with follow-ups specifically because the invented answer is easy to produce and easy to detect. Two probing questions about how the number was measured, what the baseline was, and what did not improve will expose a fabricated adoption story reliably, and being caught inventing on this topic is worse than having no experience, because it converts a gap in exposure into a question about honesty.
What works instead is to be exact about your evidence base and then reason well from it. "I have used it heavily myself and I have not run a team-wide adoption, so let me tell you what I have observed directly and then how I would approach it as a manager" is a strong opening, because it does two things at once: it establishes that you know the difference between personal experience and organisational evidence, which is the discipline the whole topic requires, and it converts the question into one you can answer well.
Then give the specifics you do have. A concrete account of one task where it genuinely saved you a day, one where the output was subtly wrong in an interesting way, and one where you chose not to use it and can say why, is more persuasive than any percentage. The subtly-wrong example is the highest-value item in that list. It demonstrates that you read the output, that you have calibration, and that you are not selling anything.
Finish with what you would do, framed as a design rather than as a plan you have executed. What you would measure, what you would refuse to measure, what you would write down before scaling, and what you would watch for on the quality side. Nobody expects a candidate to have run a controlled study. They expect a candidate to know what one would need to look like.
Two phrasings worth having ready:
"I should be straight that my experience here is individual rather than organisational. I have used these tools daily for about a year on real production code, so I have a good sense of where they help and where they mislead, but I have not been the person running a rollout across three teams. If it is useful I can tell you what I would do, and I would rather do that than dress up my own experience as something bigger."
"I have not measured this and I would not trust myself to eyeball it. What I noticed was qualitative and I think it is real: people take on the tedious jobs they used to defer. Whether that shows up in delivery data, I do not know, and I would want to look before I claimed it."
The second one is doing something specific that is worth understanding. It reports a genuine observation, labels it as unmeasured, states a belief with the confidence level attached, and declines to overclaim. That combination — observation, label, calibrated belief — is the register the whole topic rewards, and it is the same habit that separates trustworthy status reports from optimistic ones, which is why an interviewer treats this answer as evidence about how you will report on everything else.
What interviewers ask
The question arrives in every round of a modern loop, and each round is grading a different thing.
The hiring manager asks it early and conversationally, usually as "how is your team using AI" or "what has changed about how you deliver". They are grading whether you have a position that is neither of the two cheap ones, and the observable signal is whether your first answer contains a limit. A candidate whose opening sentence is entirely positive, or entirely negative, has told them which bucket they are in before the follow-up. The second signal is whether you distinguish your own experience from your team's, unprompted.
The technical panellist grades depth by pushing on a specific example. Expect "tell me about a time the output was wrong in a way you nearly missed" or "what do you not use it for". The observable signal is specificity: a candidate who can describe a concrete wrong output, why it looked right, and what caught it has read the output; one who speaks in generalities about hallucination has read articles. There is a second, quieter signal, which is whether you are still doing the engineering. An answer describing wholesale delegation of design decisions is read as a candidate whose technical judgement has thinned, which is fatal at staff level and is one of the tensions in staying technical without taking decisions off the team.
The engineering manager or director panellist grades measurement and organisational reasoning. The observable signal is whether you can name something that did not improve. This is the highest-information single moment in the whole area: a candidate who can immediately name a stage where adoption produced nothing, or produced a cost, is a candidate who looked. The second signal is whether you locate the constraint in the delivery system before making any claim about improvement.
The executive or skip-level grades commercial judgement and communication. Expect "what would you tell the board" or "our CFO thinks we can hold headcount flat, what do you say". The observable signal is whether you can be honest upwards without being an obstacle, which means giving a bounded commitment, naming the risk, and offering a way to find out rather than a position to argue about. Candidates who tell the executive what they want to hear fail this, and candidates who deliver a lecture about the limits of the technology also fail it.
A peer from another function, where present, grades whether you translate. Can you explain the review bottleneck to somebody in product without jargon, and can you explain why more code being written is not the same as more product being delivered. The observable signal is whether you reach for their units rather than yours.
One cross-cutting signal appears on scorecards in every round under different names: calibration. Whether you distinguish what you measured from what you observed from what you believe from what you read. This topic is unusually good at exposing it, because the evidence base is genuinely weak and a candidate who speaks about it with the same confidence they would use for a fact about TCP is displaying a habit that will contaminate every report they ever write. The candidates who do well tend to hedge in a very particular way: precisely, once, with the reason for the hedge attached, and then they get on with the answer.
Questions
The questions below are phrased the way interviewers put them. Each answer is followed by what the panel is grading, which is frequently not what the question appears to be about.
How is your team using AI in the delivery process?
Answer by stage rather than in general, because the general answer is a slogan. Name two or three places it lands genuinely — for most teams that is first drafts and boilerplate, test scaffolding, and documentation that previously did not exist — and then name a stage where it changed nothing, which for most teams is anything involving waiting for another team or a decision. Say what your team does not use it for and why. If you have a policy about review or data boundaries, mention it, because that tells the interviewer you have run this rather than observed it.
The grading is on whether the answer contains a limit within the first thirty seconds. The panel is sorting you into evangelist, dismisser or the group that has looked, and a stage-by-stage answer with one honest negative places you in the third immediately. Secondary credit for distinguishing what you saw at the keyboard from what you saw in delivery data.
How much faster is your team since adopting these tools?
Decline the percentage, and give something better in its place. Say that you do not have a productivity figure you would defend and briefly why: no counterfactual, everything else changed too, and the tasks people delegate are the routine ones so any comparison is between different work. Then give what you do have — a specific observable, between two defined points, with the confounders named. If you have nothing measured, say that and say what you would measure.
The graded signal is whether the refusal comes with substance attached. A candidate who says "you cannot measure that" and stops has given a lazy answer wearing rigour's clothes; one who explains the measurement problem and then offers a bounded, sourced observation has demonstrated the exact judgement the role requires. Interviewers also watch whether you sound defensive, because the same exchange happens with a CFO and composure is part of the assessment.
Our leadership wants a number for the productivity gain. What do you give them?
Give them a proposal rather than a figure. Explain that the number they would find credible in six months is not one you can produce today, then offer the instrument: a small number of outcome measures agreed before the change, a defined baseline period, a pairing of speed and quality so the report is not one-sided, and an explicit list of the other things changing in the same window. Commit to reporting what you find including if it is disappointing. If they need something now, give a qualitative statement with its confidence attached rather than a manufactured percentage.
The grading is on whether you can say no to a senior stakeholder while remaining useful, which is the same skill tested by any unreasonable request for certainty. Producing a number to satisfy the room fails hardest, because the panel knows you would be creating an obligation you cannot discharge. Refusing without offering an alternative fails almost as hard.
What metrics would you use to evaluate whether this is working?
Start with what you would not use and why: lines of code, pull request counts, acceptance rate and percentage of code generated all measure activity or engagement rather than delivered outcome, and all are gameable without anyone intending to game them. Then propose outcome measures — cycle time as a distribution rather than a mean, deployment frequency, change failure rate, time to restore — with the explicit caveat that these are guardrails rather than attribution. Add a breakdown of where a change waits, because that is what tells you whether relieving authoring could have helped. And pair every speed measure with a quality measure.
Two things are graded. Whether you volunteer the attribution limit unprompted, because a candidate who presents DORA as proof of an AI effect has misunderstood what the measures are. And whether you name a quality measure at all, since watching only the speed pair is watching the half of the instrument that can produce the answer you want.
Why not just measure lines of code, or commits, or pull requests?
Because all three measure production of artefacts, and the artefacts became nearly free. Lines of code was already inverted before this — the best change is often a deletion — and it now tracks the tool's verbosity. Pull request counts measure batching convention, so a team that splits its work more finely doubles the number without changing output. Commits measure habit. Beyond being weak, they are actively harmful once used for evaluation, because they change behaviour in a direction unrelated to value, and everybody involved knows how to move them.
The grading is on whether you explain the mechanism rather than just asserting that the metric is bad. Anyone can say lines of code is a poor metric; the candidate who says the marginal cost of a line approached zero so the metric now measures the generator rather than the engineer has demonstrated that the objection is reasoned.
Where in your delivery process did it help least?
Anywhere the constraint was waiting rather than working. Cross-team dependencies, shared environments, release windows, decisions from people who are not in your team, and anything requiring a stakeholder to make up their mind. Also, in most codebases, work in the oldest and most idiosyncratic parts of the system, because a generator works from patterns and a fifteen-year-old system's conventions are out of distribution. And comprehension work: reading and understanding an unfamiliar subsystem is not meaningfully accelerated, and it is a large share of the job.
This is one of the highest-information questions in the set and it is graded on whether you have an answer at all. Every candidate can name where it helped. A candidate who can immediately name where it did not has looked at their own process rather than at their own experience, and that distinction is close to the whole assessment.
Has your review process changed?
It has to have changed, and if it has not, that is the honest answer and the interesting part is why. What typically changes: pull request size limits become more important rather than less, because the cost of producing a large change fell and the cost of reviewing one did not. Automated checks take over the mechanical layer so human attention goes to system fit, duplication and unstated invariants. Authors are explicitly accountable for explaining anything they submit. And reviewer capacity gets watched as a first-class constraint rather than assumed.
The graded insight is whether you understand that review is now the binding constraint. A candidate who describes faster generation with an unchanged review process has described a growing queue and not noticed, and the follow-up will be about what happened to their review turnaround. Secondary credit for observing that reviewers lose their heuristics with generated code, because the usual cues about where the author was unsure are gone.
Your review queue has grown since adoption. What do you do?
Establish first whether the queue grew because more change is arriving or because reviews are taking longer, since those need different responses. If arrival rate rose, the options are to raise review capacity, reduce what needs human review by strengthening automated gates, or reduce batch size so each review is cheaper. If review is slower, find out why — often it is because reviewers are being handed larger and less familiar diffs. Then decide explicitly whether you are willing to accept the queue, because an unmanaged queue silently converts into shallower reviews rather than into a longer wait.
The graded behaviour is treating it as a flow problem with a diagnosis step, rather than reaching immediately for "we need more reviewers". The second signal is naming the silent failure: under load, reviews get shallower and nobody announces it, so the metric to watch is not queue length but escaped defects.
Do you worry about quality? What have you seen?
Yes, and be specific about the character of the risk rather than gesturing at hallucination. The defect profile shifts: fewer typos, more plausible-but-wrong boundary assumptions, more code that works for the example and not at production volume, and — the important one — more code that is subtly inconsistent with an invariant that holds everywhere else in the system but is written down nowhere. That last class is invisible in a diff by construction, because a diff shows the change and not the seventeen other places that assumed something different.
The grading is on whether you can describe a failure mode that is mechanically explicable. Candidates who say "it hallucinates" have read about the problem; candidates who describe a specific class of error and why review is poorly adapted to catching it have hit the problem. If you can add what you changed in response, you have answered a level above the question.
What is your view on AI-generated tests?
Useful for scaffolding, fixtures, parameterised cases and coverage of obvious branches, and dangerous as the specification. A generator writing tests against existing code writes tests asserting what the code does rather than what it should do, so pointing it at a function containing an off-by-one produces a suite that permanently ratifies the off-by-one and makes it harder to fix, because fixing it now breaks tests. The workable split is that tests encoding intended behaviour are written by a person from the requirement, ideally before the code, and generated tests are for regression pinning around them.
This is a depth probe and the off-by-one mechanism is the answer that scores, because it is concrete, checkable and explains itself. A secondary signal is whether you note that coverage as a target has become even less meaningful now that the marginal cost of a test is near zero.
Who owns code that a model generated?
The person who submitted it, entirely, with no diminished category. That means they have read every line, can explain why it works, and would have been prepared to write it. When it fails in production, the tool is not an available explanation. Make it explicit rather than assumed, because a stated norm changes behaviour and an implied one does not.
The grading is on speed and firmness. This question has a correct answer and candidates who hedge towards shared or diminished ownership are describing a codebase with unowned regions in it. Additional credit for having a view on provenance marking, particularly for saying where it is genuinely required — regulated environments with audit obligations — and where it risks creating the two-tier ownership you just ruled out.
What are you not allowed to send to these tools, and how do you enforce it?
Name the categories: customer and production data, secrets and credentials, source code where the client contract or the security policy forbids egress, and anything subject to residency constraints. Then describe enforcement as a control rather than as training, because developers paste things and a policy that assumes they will not is not a control. In practice that means an approved tool list with the terms checked, pre-flight scanning where you can place it, network controls where the environment allows, and a per-client default of not permitted until confirmed. Have an incident process for when a secret goes out, because it will, and the response is rotation rather than embarrassment.
The graded signal is whether you treat the constraint as legitimate rather than as friction. A candidate who describes arguing for exceptions has told the panel how they will behave with a security reviewer. The second signal is per-client differentiation in a services context, which is the detail only somebody who has run it will volunteer.
How do you handle the licensing risk in generated code?
Accept that you cannot fully verify provenance of output and put the control in the pipeline: licence and dependency scanning on every change, with a policy about which licences are acceptable in which products. Check what the vendor agreement says about indemnity and about whether your input trains their model, because those terms vary and they are the part that legal will ask you for. Then be clear about the boundary of your competence — the copyright position is unsettled and jurisdiction-dependent, it is owned by legal rather than by you, and your job is the engineering control while it remains open.
The grading is on knowing where your authority ends. A candidate who confidently states the legal position has overstepped; one who says "I do not know and here is who does, and here is the control I run regardless" has behaved the way you want an architect to behave on any unsettled question.
What does this do to junior engineers?
Name the mechanism rather than the anxiety. Skill forms through struggle — being stuck, being wrong, and building a mental model out of the residue — and a tool that removes the struggle removes the mechanism at exactly the stage it matters most. The observable symptom is a correct pull request the author cannot explain, which does not surface through normal channels because the pull request is fine; it surfaces during an incident or in a promotion discussion eighteen months later. Then give the countermeasures: explanation as a routine part of review, certain categories of work kept deliberately unassisted because comprehension is the deliverable, pairing on the hard problems, and progression criteria that assess depth rather than output.
The grading is on whether you propose something workable rather than either dismissing the concern or proposing a ban. Banning the tools for juniors is impractical and slightly absurd, since it is what the industry uses. The candidate who reframes it as deliberate practice design is answering as a manager rather than as a worried senior engineer.
How would you interview an engineer now, given they will use these tools in the job?
Move the assessment towards what cannot be delegated. Reading unfamiliar code and explaining what it does and where it is fragile. Debugging something broken with a real symptom. Reviewing a change and finding the problem in it. Defending a design decision under questioning about constraints that emerge during the conversation. Allow the tools where the job allows them, and grade the judgement about when to use them, since that judgement is now part of the craft. Stop grading whether somebody can produce a working function from a specification, because that is the part that got cheap.
The signal is whether you redesign the assessment or police the candidate. Forbidding assistance tests a condition the job does not contain and is largely unenforceable in a take-home. The second signal is whether you name the thing you would now weight higher, which for most roles is explanation depth under follow-up.
One of your senior engineers refuses to use these tools. What do you do?
Find out what the objection is, because there are several and they need different responses. A quality objection is testable: ask what would have to be true for them to use it, and you have converted a position into a proposition. A review-load objection is legitimate and probably a signal about your process rather than about them. An ownership objection — they do not want to be accountable for code they did not reason through — is the correct instinct and deserves a policy answer rather than persuasion. And an identity objection, where the discomfort is about what the work is becoming, is real and needs a conversation rather than a mandate. Do not require usage; require the outcomes, and let them meet them however they meet them.
The graded behaviour is treating a competent senior's scepticism as information rather than as non-compliance. Candidates who reach for a mandate have described how to lose a good engineer and produce compliance behaviour from everyone else. Secondary credit for noticing that some refusals are about review load, which makes it your problem rather than theirs.
Your company has frozen headcount on the assumption of an AI productivity gain. You do not think it holds. What do you do?
Disagree specifically and in writing, then execute. Specifically means naming what you can commit to and on what basis, rather than arguing about the premise: here is the throughput I will commit to, here is what it assumes, here is what I would need for the larger number, and here is the leading indicator that will tell us within a quarter which way this is going. Propose the review point up front so the decision can be revisited with data rather than reopened with opinion. Then get on with it, because relitigating a decision that has been taken makes you a management problem regardless of whether you were right.
The grading is on whether you can disagree without becoming an obstacle, which is the general senior-management skill this scenario is chosen to test. Quiet compliance fails because you have accepted a commitment you believe is undeliverable. Open resistance fails because it makes the disagreement about you. Naming an early indicator and a review point is the move that scores, because it gives everybody a way to be wrong cheaply.
How do you roll this out across several teams?
Treat it as a practice change rather than a procurement. Start with volunteers on real work, give them time rather than encouragement, and require them to report the failures as well as the successes, because the honesty is what makes the report credible to the people who are not convinced. Write the norms down before scaling — what review looks like, what may be sent where, who owns the output — and have engineers own those norms rather than issuing them. Budget the capacity the adoption will consume in the quarter before it returns anything. Then scale on evidence, and be willing to say that a team where it is not helping should stop.
The graded elements are not mandating usage and budgeting the cost. A mandate produces compliance behaviour and burns the credibility of the next change programme. An unfunded adoption comes out of delivery capacity nobody planned for, and somebody will be surprised in a steering meeting. Candidates who mention that one team may reasonably conclude it does not help them have shown they are running an experiment rather than a rollout.
Which parts of the lifecycle should not be automated?
Anything where the output is a decision rather than an artefact. Deciding what to build. Choosing between designs under constraints that are not written down. Deciding to roll back during an incident. Accepting a risk. Judging whether a change is right for this system given its history. Those are not automation candidates because they require accountability, and accountability cannot be delegated to something that cannot be held to it. Separately, some things should stay unassisted for developmental reasons rather than technical ones, which is a different argument and worth distinguishing.
The signal is whether the line you draw is principled rather than a list. "Decisions, not artefacts" is a line that generalises to a tool you have not seen yet, which is the property an interviewer is testing for when they know the tooling will change again before you finish onboarding.
Give me an example where the output was wrong in a way that nearly got through.
Give a real one with enough specificity that it is checkable. The strongest examples share a shape: the code was idiomatic and plausible, it passed the tests that existed, and it was wrong about something that lived outside the diff — a convention held elsewhere in the codebase, an assumption about a boundary condition, an error case that could not occur alongside a missing one that could, or behaviour that was fine at test volume and not at production volume. Say what caught it, and be honest if what caught it was luck.
This is graded almost entirely on whether the example is real, and interviewers will ask a second and third layer of detail to find out. The follow-up is usually what you changed afterwards, and "nothing, we were careful" is a weaker answer than a small concrete change to a checklist or a test practice. Admitting that luck caught it scores better than a tidy story, because it is the answer somebody gives when they are not managing their image.
What would you tell the board about AI and engineering?
Give them the three-claim separation in plain language: generation is genuinely faster, delivery throughput depends on where your constraint is, and business outcome depends on whether the extra capacity is spent on something that matters. Say what you are doing, what you expect, and how you will know. Name the risk you are managing — review capacity, defect escape, data boundary — because a board that hears only upside will not believe the next report. And decline to give a percentage, while giving them the date by which you will have real evidence.
The graded skill is translation without dilution. An answer full of engineering vocabulary fails; an answer that promises a transformation fails differently and more expensively, because it creates an expectation you will be held to. Naming the risk unprompted is what makes the upside credible, and boards are unusually good at noticing when they have only been told good news.
Do you think this changes what an engineer is for?
Have a view and hold it lightly. The defensible position is that the centre of the job moves further towards the parts that were always the hard parts: understanding a system well enough to change it safely, deciding what is worth building, judging whether something is correct, and owning the consequence. Producing text that a compiler accepts was never the essence of the work; it was the medium. What genuinely changes is the ratio, and the ratio shift is uncomfortable because writing was the part that felt like progress and verifying is the part that feels like overhead.
The grading is on whether you can discuss a large open question without either grandiosity or defensiveness. A candidate who declares the profession transformed has overclaimed on a question nobody can answer; one who says nothing has changed has told the panel they are not paying attention. The composed answer notices that the question is genuinely open and says which way they lean and why.
You have not run this at scale. What would you do in your first ninety days here?
Find out where change actually waits before touching anything, because you cannot know whether faster authoring helps until you know what fraction of elapsed time authoring was. Establish a baseline on a small number of outcome measures and write it down while it is still uncontaminated. Find the people already using it and ask them what went wrong rather than what went well. Read the data-boundary constraints, because in some organisations that is the whole answer and it is better to learn it in week two than in month four. Then run one deliberate change with a stated hypothesis, and report the result honestly whichever way it goes.
The graded element is sequencing: baseline before intervention, and diagnosis before prescription. A ninety-day plan that begins with a rollout has told the panel you will optimise something without checking whether it is the constraint. Candidates who put the data-boundary review early have usually worked somewhere it mattered, and that detail reads as experience more reliably than any claim about outcomes.
What has this changed about how you personally work?
Answer concretely and include a cost. The honest version for most engineers is that first drafts arrive faster, unfamiliar territory is less intimidating, tedious jobs get done rather than deferred, and reading has gone up — you spend more of the day evaluating text you did not write, which is a different and more tiring activity than writing. Some people find that trade good and some find it draining, and saying which you are is more interesting than saying it is fine.
The grading is on candour and self-observation. This question is often the last one and it is not filler; it is a check on whether the considered answers you gave earlier came from experience or from preparation. An answer that is entirely positive fails the same way the opening answer would, and a specific personal cost named without complaint is the strongest thing you can say.