An auditor wants proof that a control operated on every day of the review period, not that it exists today. What do you show them?
Evidence of operation is a by-product of the control running, not a screenshot taken during the audit. Design each control so it emits a dated, attributable artefact you cannot backfill, and be able to argue completeness rather than just show samples.
What the interviewer is scoring
- Does the candidate distinguish evidence that a control exists from evidence that it operated
- Whether evidence is designed as a by-product of the control rather than assembled during the audit
- That completeness is argued from the population, not from a handful of favourable samples
- Can the candidate name a control they run whose only evidence is somebody's memory
- Whether the candidate says plainly what happened on the days the control did not operate
Answer
Existence and operation are two different claims
The question separates candidates faster than almost anything else in this area, because most preparation is about the first claim and every audit of consequence is about the second. Existence is easy: here is the policy, here is the pipeline configuration requiring two approvals, here is the screenshot of the alerting rule. All of that is a statement about the present tense, and the auditor is asking about the past. A report covering a period asserts that controls operated throughout it, which means the object being tested is not your configuration but a population of events stretching backwards, and the only thing that can satisfy it is an artefact created at the time.
This is why evidence gathered during audit week is structurally weak. It shows the control was in place on the day somebody looked. It says nothing about the fortnight in the middle of the period when the job was failing silently, or the release that went out through an emergency path with the approval requirement bypassed, and an auditor whose whole professional instinct is to test rather than accept will go looking for exactly those.
Design the control so it leaves a trail it cannot help leaving
The useful reframing is that evidence should be a by-product, not a deliverable. If producing evidence is a separate activity somebody has to remember, then the evidence and the control fail independently and you will eventually have one without the other. If the control cannot execute without emitting a dated, attributable record, the two are welded together and the absence of a record becomes meaningful — a gap in the trail is itself a finding rather than an administrative oversight.
That distinction produces a concrete test you can apply to any control you run. Ask what artefact exists, today, dated inside the period, that could only have been created by the control actually running. If the answer is a policy document, a wiki page or a colleague's recollection, you do not have evidence of operation.
| Control | What teams typically offer | What evidences operation |
|---|---|---|
| Independent approval before production change | The branch protection settings | Per-release approval records with approver identity, plus the list of releases that bypassed the path |
| Access reviews | The review policy and a spreadsheet template | Dated completed reviews with the reviewer named and the resulting revocations traceable to the review |
| Joiner-mover-leaver | The HR process description | Correlation of termination dates against credential disablement times, including the outliers |
| Backup restorability | The backup job's green dashboard | A restore performed to a separate environment, its date, its outcome and who observed it |
| Vulnerability remediation | A current scan showing nothing critical | The queue over time, with age at closure and documented decisions on what was accepted |
| Log monitoring | The alerting configuration | Alerts that fired, who acknowledged them and what was done, including the ones closed as noise |
The right-hand column shares a shape. Each entry is dated, attributable to a person or an identity, and produced by the ordinary operation of the system rather than by an audit-preparation exercise. When an interviewer wants to know whether you have done this before, that shape is the tell.
Five backup mechanisms and no restore
In January 2017 a GitLab.com engineer working on replication late at night ran a deletion against the wrong host and removed the production database directory, then discovered that five separate backup or replication mechanisms had all failed to leave a usable, recent restore. Several hours of data were lost permanently.
Read as an evidence problem rather than an operations one, this is the whole argument in a single incident. Five controls existed. Each would have passed an existence test comfortably — configured, scheduled, green. None had recently been exercised end to end, and the difference between a backup job completing and a restore succeeding is precisely the difference between the two claims. So the questions that matter for a control like this are when the restore was last actually performed, to where, whether the restored data was verified rather than merely mounted, and who other than the person running it saw the result. A control whose success signal comes from the same code path as the control itself is not a control, it is a self-report.
Arguing completeness, not just producing samples
The subtler half of the answer concerns the population. An auditor selects samples from a population you supply, and the integrity of the whole exercise depends on that population being complete. If you hand over the changes that went through the standard pipeline, the sample is drawn only from the well-behaved subset, and the emergency changes — the ones with the interesting approval story — are invisible because they never entered the list.
So the stronger move is to derive the population from the system of record for the thing itself rather than from the control's own log. Deployments should be enumerated from what is running in production, not from the pipeline's success table. Leavers should come from the HR system, not from the tickets that were raised. Then reconcile: every item in the authoritative population either has control evidence or is an exception, and you name the exceptions yourself. Volunteering a known gap costs a finding. Having one discovered after you presented a clean population costs your credibility on every other assertion, and that is the more expensive outcome by a wide margin.
Who could have edited the evidence
Expect to be pushed on the integrity of the artefacts. Evidence that sits in a system where the people being evidenced can edit history is weaker than evidence in an append-only store with a separate administrative boundary, and it is worth being honest about where yours sits. Cryptographic chaining, write-once retention on the object store and shipping to an account nobody in the engineering organisation can administer all raise the bar; none of them makes the record tamper-proof, because whoever controls the writing path controls what gets written in the first place. What they achieve is tamper-evidence, and the reason that is enough is that it changes an undetectable edit into a detectable one.
Retention deserves the same honesty. Evidence has to outlive the period it covers by long enough to be examined, and different obligations pull in different directions — a retention requirement may hold an artefact that a privacy obligation would otherwise have you remove. Where the specific durations come from, and how they are reconciled, varies by jurisdiction and by industry, and it is a decision for your legal and compliance function rather than one to assert from an architect's chair. What you can insist on is that the duration is a deliberate choice recorded somewhere, rather than whatever the default log rotation happened to be.
Ask of every control you own: what artefact, dated inside the period, could only exist because the control ran? If the answer is a policy document or somebody's memory, you have a control with no evidence, and audit week is far too late to discover which ones those are.
Likely follow-ups
- Your evidence for a control lives in a ticket system anyone can edit. Does that weaken it, and how far?
- A control failed for three weeks in the middle of the period. What do you do before the auditor finds it?
- How would you evidence a control performed by a third party you cannot instrument?
- What is the smallest change to your pipeline that would generate audit evidence you do not currently have?
Related questions
- How do you design a system so an auditor can prove who accessed a record and why?mediumAlso on tamper-evidence8 min
- What is the difference between SOC 2 and ISO 27001, and what does an auditor accept as evidence?mediumAlso on audit-evidence6 min
- An upstream feed resets ten thousand of your prices to a penny and orders start arriving. What should have stopped it, and what do you do with the orders that got through?hardSame kind of round: scenario6 min
- A release goes out and afterwards users can see records they were never entitled to. How would you find out, and how long would it take?hardSame kind of round: scenario6 min
- A configuration change that passed review takes the entire fleet down in under a minute. What in your design should have limited the damage?hardSame kind of round: scenario6 min
- A maintenance script updates your permissions table and every user in one tenant can suddenly edit everything. What should have stopped that, and how would you have found out?hardSame kind of round: scenario6 min
- A Terraform plan in CI wants to destroy resources in production. What has to happen before anyone is allowed to approve it?hardSame kind of round: design6 min
- A team wants to add a third-party AI provider hosted in another country to a feature your European customers use. What has to be true before that first request goes out?hardSame kind of round: scenario6 min