How do you decide what to automate and what to leave manual?

The decision is per test case, not per feature

The question is usually asked as if automation were a property of a feature, and the useful move is to reject that framing straight away. Automation is a property of an individual check, and a single user story will normally split across three or four layers plus a handful of things you will never automate at all. Teams that decide at feature granularity end up with a UI test for a rounding rule, which is the most expensive possible way to assert arithmetic.

Four criteria do most of the work. Execution frequency is the first, because an automated test only pays back if it runs: something exercised on every commit earns automation quickly, while a check that runs once per major release probably does not. Determinism is the second, and it is really about whether an oracle exists. If you can state the expected result as a value rather than a judgement, a machine can assert it; if the expected result is "the report looks reasonable" or "the layout does not feel broken", no assertion you write will mean what you want it to mean. Cost of the bug escaping is the third, and it is what justifies expensive tests: a payment double-charge or a permissions leak deserves an end-to-end check even though end-to-end checks are the ones you least enjoy owning. Stability of the surface under test is the fourth and the most neglected. Automating against an interface that is still being redesigned means you are paying to rewrite tests, not to find bugs, so a deliberate lag between a UI settling and its regression pack being written is good judgement rather than laziness.

Pyramid, trophy, and why UI-heavy suites rot

The pyramid says put most checks at the unit level, fewer at the integration or service level, and very few at the end-to-end level, because cost and runtime and fragility all rise as you climb. The testing trophy rebalances that for applications where most real defects live in the seams: it keeps unit tests but makes the integration layer the widest band, on the argument that a service tested through its own API with real collaborators catches wiring bugs that isolated unit tests with mocks cannot see. Both shapes encode the same principle, which is the only part worth defending in an interview: push each check to the cheapest layer at which it can still fail for a real reason. The shape you end up with is an output of that rule applied to your architecture, not a target to draw first.

UI-heavy suites rot for reasons that compound. Every test depends on the whole system, so any instability anywhere shows up as a failure here, and the failure gives you a screenshot rather than a stack trace pointing at a line. Diagnosis cost per failure is high, so a red build gets rerun instead of read. Because each test drives the app through the same login-and-navigate path, a single change to that path breaks hundreds of tests at once, and the cheapest response is always to loosen an assertion or add a wait. Do that a few dozen times and you have a suite that takes an hour and asserts almost nothing, which is worse than having no suite because it still consumes trust and CI time.

Flakiness is a cost, not an annoyance

Treat a flaky test as a defect with an owner and a ticket. Its real cost is not the rerun, it is that a suite failing intermittently teaches the team to ignore red, and once that habit forms the genuine regression it eventually catches will also be ignored. The practical policy is: quarantine on detection so the pipeline signal stays clean, fix or delete within a fixed window, and never let a blanket global retry stand in for a root cause. Retries are legitimate as a narrow, logged mitigation for a known infrastructure flake; they are illegitimate as a default, because they convert a real intermittent product bug, such as a race condition, into a green build.

Exploratory testing is genuinely not automatable

Automated tests only ever check things someone already thought of, which makes the suite a regression net rather than a discovery tool. Exploratory testing is the discovery activity: a tester works a time-boxed charter over an area, forming and testing hypotheses about how it might break, and finds the problems nobody wrote a case for. It is the only way to catch defects of omission, confusing behaviour that is technically correct, and the awkward interactions between two features that were specified separately. Budget it as scheduled work with charters and notes and follow-up bugs. When exploration finds something important and repeatable, that is exactly when you write the automated test for it, so exploration becomes the feed into the suite rather than its residue.

A worked example

Take "apply a promotional code at checkout", with rules for percentage and fixed-value codes, expiry, minimum basket value, one use per customer, and non-stacking.

Check	Layer	Why there
Discount arithmetic, rounding, expiry boundary, minimum-basket boundary	Unit	Pure logic with an exact expected value; cheap and runs on every commit
One-use-per-customer enforcement under concurrent redemption	Integration, against a real database	The bug lives in the constraint and transaction, not the calculation
Non-stacking rules across code combinations	Integration or service API	Combinatorial, so it needs the fast layer to be affordable
Discount response contract between checkout and pricing service	Contract test	Catches provider schema drift without a full environment
One happy path: valid code applied, total updates, order places at the discounted amount	End-to-end UI	High escape cost, and it is the only layer that proves the pieces are wired together
Invalid-code error message wording, placement, and screen-reader announcement	Manual, with visual review	The oracle is human judgement about clarity
Charter: "try to get a discount you should not be entitled to"	Exploratory	Discovery, not regression

The shape falls out on its own. Dozens of unit assertions, perhaps ten integration tests, one contract test, exactly one UI test, and two named human activities. Note what is not there: no UI test per code type, and no automated assertion on error copy that product will reword next sprint.

The trap

The trap is answering with a coverage target. Candidates say "we aim to automate eighty percent of regression cases" and think it sounds rigorous, but it is the exact mechanism that produces rotting UI suites, because the cheapest way to hit a case-count target is to automate whatever already has a written manual script, at the layer the script was written for. It also implies the remaining twenty percent is a failure to be closed, which quietly deletes exploratory testing from the plan. The strong answer states criteria, applies them per check, and is comfortable saying that some cases should never be automated and some existing automated tests should be deleted.

Automation is a decision per check against stated criteria, not a percentage to hit; the strongest signal you can give is naming the cheapest layer at which each behaviour can still fail, and admitting which checks belong to a human instead.