Regulation and compliance fundamentals

A regulation is law — GDPR, HIPAA, SOX — enforced by a state authority with the power to fine you. A standard is a published specification you can be certified against, such as ISO 27001 or PCI DSS, and its force is contractual: a customer or a card scheme requires it, not a legislature. A framework is a structured set of control objectives you map yourself onto, such as NIST CSF or COBIT, with no certificate at the end. A policy is your own organisation's rule, written internally, and the only one of the four you can change unilaterally. The distinction decides who you can negotiate with. A policy can be amended by an internal decision, a standard is a commercial conversation, and a regulation is a constraint you design around — so establishing which one you are facing is the first useful question when told something is not allowed.

Because auditors test controls against objectives and rarely prescribe implementations. "The auditor requires ninety-day password rotation" is almost always somebody's summary of an objective about credential compromise, which your organisation chose to satisfy by rotation some years ago. That matters because the objective usually admits a better implementation — phishing-resistant multi-factor authentication, short-lived credentials issued per session — and you can only propose one if you can name the objective the existing control serves. The habit worth building is to ask which control a request maps to and which objective that control satisfies. When nobody can answer, you are complying with a policy that has outlived its reason, and changing it is paperwork rather than an argument with a regulator.

Compliance work is this trace repeated a few hundred times, and an engineer who can produce one row of it unprompted is immediately credible.

objective                      control                          evidence accepted
-----------------------------  -------------------------------  ------------------------------------
Only authorised changes        Every production change has a    Change record with PR link,
reach production               reviewed merge request and a     approver identity, timestamp, and
                               named approver who is not the    the pipeline run that deployed it
                               author

Access is removed when         Leaver process revokes all       Ticket from HR, dated access
someone leaves                 accounts within 24 hours         removal log per system, quarterly
                                                                reconciliation of active accounts
                                                                against the HR roster

Cardholder data is not         PAN is tokenised at the edge     Data-flow diagram, a scan of the
retained by the application    and never persisted by us        application datastore showing no
                                                                PAN, plus the vault's attestation

Backups can actually be        Quarterly restore test into an   Test record with date, operator,
restored                       isolated environment             restored dataset, measured RTO,
                                                                and the defects raised

Read the third column first, because it is the one engineers underestimate. An auditor is not testing whether your intent is good; they are testing whether an independent person can verify the control operated throughout the period. A control that works perfectly and leaves no artefact fails the audit, and that is not pedantry — an unevidenced control is indistinguishable from one nobody ran.

The second thing the table shows is that evidence is generated, not gathered. Every row above is a by-product of a system doing its job: the pipeline already knows who approved the merge, the identity provider already knows when the account was disabled. The expensive version is the one where a human assembles screenshots at the end of each quarter, because that scales linearly with scope and breaks the first time somebody is on holiday.

Third, notice that two rows include a reconciliation rather than only a log. A log tells you what your process did; a reconciliation tells you whether reality matches, and it is the only thing that catches the account created outside your process. Auditors ask for it because it detects the control being bypassed rather than failing.

The design consequence is to treat evidence as a functional requirement. When building the release pipeline, the access system, or the deletion job, ask what artefact it emits, where that artefact is retained, and for how long — because retrofitting it later means reconstructing a period you no longer have data for.

Compliance is meeting a defined set of controls that somebody else wrote, assessed at a point in time against a scope you agreed. Security is the actual property of being difficult to attack. They overlap substantially and neither implies the other. A system can pass every PCI requirement and be trivially exploitable through a business-logic flaw nobody wrote a control for, and a genuinely secure system can fail an audit because it cannot produce records. The useful framing in an interview is that compliance sets a floor and defines the language you use to argue for security spending, while the threat model is what tells you where the real risk is. Treating the audit report as a security assessment is the mistake that produces breached, certified companies.

The register is where compliance and engineering actually meet, because the residual column is a decision somebody has to sign.

Risk ID                 R-114
Risk                    Customer PII exposed through an over-permissive
                        object-storage bucket
Owner                   Head of Platform
Inherent likelihood     4 of 5      (misconfiguration is common and easy)
Inherent impact         5 of 5      (regulatory notification plus contract breach)
Inherent rating         20          HIGH

Controls                1. Buckets default to private, enforced by policy-as-code
                        2. CI fails on any public-access grant
                        3. Daily scan of live configuration with alert
Control effectiveness   Assessed EFFECTIVE, tested 2026-05-12

Residual likelihood     2 of 5      (requires bypassing CI and the daily scan)
Residual impact         5 of 5      (unchanged - impact is rarely reduced by
                                     preventive controls)
Residual rating         10          MEDIUM
Treatment               ACCEPT, reviewed quarterly
Accepted by             CTO, 2026-05-20

The single most instructive line is that impact did not move. Preventive controls reduce likelihood; only a change to the data itself — holding less of it, holding it tokenised, shortening its retention — reduces impact. Candidates routinely claim a control lowered both, which tells an experienced interviewer that the model is being filled in rather than reasoned about.

The second point is that residual risk is accepted by a named person, not by the engineering team and not by the register. This is the mechanism by which an organisation converts a technical fact into an owned business decision, and it is why raising a risk formally is more effective than raising it in a stand-up. If the answer is "accept", you have documentation; if the answer is "that is unacceptable", you have funding.

Third, the treatment options are a fixed vocabulary — mitigate, transfer, accept, avoid — and knowing them keeps a conversation productive. Insurance is transfer. Removing the feature is avoid. Most engineering work is mitigate, and most arguments happen because someone wants mitigate and someone else has quietly chosen accept.

The engineering reading is that a risk register is a prioritisation tool you can use. A finding with a residual rating and a review date beats an architectural concern raised in a document nobody signed.

It is the recognition that your compliance obligations do not stop at your perimeter: if a sub-processor loses your customers' data, you are the one who notifies the regulator. Practically this means every vendor holding regulated data needs a due-diligence record, a contract with data-protection terms, an identifiable location for the data, and a place in your register of sub-processors. The engineering consequences are concrete — you cannot adopt a new analytics SDK unilaterally, an integration must log what it sends, and any vendor in a critical path needs a documented exit plan, because concentration on a supplier you cannot leave is itself a finding in regulated sectors.

Six things that show up in code rather than in a policy document. You must have a lawful basis for each processing purpose, which means purposes have to be enumerable. You must collect the minimum data for that purpose and not silently reuse it for another. You must be able to find every piece of personal data about one individual across every system, in order to answer access, rectification and erasure requests within a month. You must delete data when its retention period ends rather than keeping it forever. You must be able to detect and report a breach within seventy-two hours. And you must know where the data physically lives. Each of those is an architectural constraint, which is why privacy work lands on engineers rather than lawyers.

It is the legal justification for processing personal data — consent, contract, legal obligation, vital interests, public task, or legitimate interests — and you need one per purpose, decided before you process. It changes the schema because different bases carry different rights: data processed under consent must stop when consent is withdrawn and the consent event itself must be recorded with timestamp and version, while data processed to fulfil a contract cannot be erased on request while the contract runs. A system that stores a single boolean for "marketing okay" has already lost the information needed to answer an auditor, because the question is which purpose, when, and on the basis of what wording.

Minimisation means you collect and retain only what the stated purpose needs, so a signup form asking for date of birth without a reason is a defect rather than a nice-to-have. Purpose limitation means data collected for one purpose cannot be reused for another incompatible one without a fresh basis — the support ticket history cannot quietly become training data for a model. Together they push against the default engineering instinct to log everything and keep it in case it is useful. The practical implementation is a data inventory that records purpose per field, which is also what makes retention and subject access answerable rather than a discovery exercise every time.

Anything relating to an identifiable living person, directly or in combination with other data you hold — so it includes the obvious name and email, and also IP addresses, device identifiers, cookie IDs, precise location, and a user ID that maps to a person in another table. The combination clause is the one candidates miss: a table of "anonymous" event data keyed by session ID is personal data if any system of yours can join session to account. Special categories — health, biometrics, ethnicity, sexual orientation, political and religious belief, trade union membership — carry stricter conditions, which is why inferring health status from purchase history is a serious design decision rather than a feature.

Pseudonymised data has direct identifiers replaced by a token, but a mapping exists somewhere that can reverse it, so it remains personal data and stays fully in scope. Anonymised data cannot be re-identified by any reasonably likely means, so it falls outside the regulation entirely. The gap between them is where most claims collapse: hashing an email is pseudonymisation, not anonymisation, because anyone with a candidate email can compute the hash and confirm a match. Genuine anonymisation requires removing or generalising enough that no combination of quasi-identifiers singles anyone out, which usually means aggregation and a check on small group sizes. Claiming anonymisation while retaining a join key is the most common privacy overstatement in engineering.

The controller decides why and how personal data is processed; the processor acts only on the controller's documented instructions. Your SaaS product is usually the processor for your customers' data and the controller for your own employee and prospect data, and it is common to be both at once for different datasets. The engineering consequences are real: as a processor you must not decide to use customer data for your own purposes, including model training, without the controller's instruction; you must maintain a list of sub-processors and notify changes; and breach notification runs to the controller rather than to the regulator. Getting the role wrong makes every downstream obligation land on the wrong party.

One person asks for everything you hold about them, and the answer reveals how many copies your architecture has quietly created.

Request: SAR-2231, verified identity, due within one calendar month

source system            what it held                        how it was found
-----------------------  ----------------------------------  -----------------------
accounts DB              profile, addresses, phone           by user_id
orders DB                order history, delivery notes       by customer_id
payments vendor          card token, last four, receipts     vendor API by ref
support desk             ticket bodies with free-text PII    text search, manual
                                                             review for third parties
email platform           campaigns sent, opens, clicks       by hashed email
analytics warehouse      event stream keyed by session       session-to-user map
                                                             table joined back
log archive              request logs containing IP and      time-bounded grep over
                         user agent for 90 days              cold storage
CRM                      sales notes written by staff        by account, redaction
                                                             of internal opinion

The two hard rows are the last three. Free-text fields contain other people's personal data — a support ticket that mentions a colleague — and you must disclose the requester's data without disclosing anyone else's, which forces human review that cannot be automated away. Internal notes and opinions may be partially exempt, but the exemption has to be applied deliberately and recorded.

The analytics row is the architectural lesson. Event data was designed as "anonymous" and is not, because a mapping table exists, so it is in scope for disclosure and for erasure. That mapping table is the difference between a system you can answer for and one you cannot, and it was created by an engineer who wanted funnel analysis.

The log archive shows why retention is a privacy control rather than a cost control. Ninety days of logs is a bounded search; five years of logs in an undocumented bucket is a request you cannot complete honestly.

What a strong answer names next is the machinery: a per-subject discovery service that queries every registered store by a known key, a registry that every new datastore must be added to, and a rehearsal, because the first SAR should not be the first time the pipeline runs. The one-month clock does not pause while you build it.

Erasure is the request that exposes architecture, because two common design choices — immutable logs and immutable backups — appear to make it impossible.

flowchart TD
    R[Erasure request verified] --> H{Legal hold or<br/>retention obligation?}
    H -->|yes| K[Retain the minimum,<br/>record the exemption and tell the subject]
    H -->|no| P[Delete from primary stores]
    P --> D[Purge derived copies:<br/>search index, caches, warehouse, CRM]
    D --> E[Event log: append a tombstone and<br/>crypto-shred the per-subject key]
    E --> B[Backups: add to a suppression list<br/>applied on every restore]
    B --> C[Confirm to the subject within one month]

Three things genuinely cannot be erased on request, and naming them is what separates a real answer from a confident one. Data you are legally required to keep — invoices for tax, transaction records under financial regulation — is exempt, and the correct response is partial erasure plus an explanation, not refusal. Immutable backups cannot be rewritten without destroying their integrity and their own audit value. And an append-only event log cannot have history removed without invalidating every downstream projection built from it.

The alternative for all three is the same mechanism: crypto-shredding. Encrypt each subject's personal fields with a per-subject key, keep the keys in a separate store, and delete the key on erasure. The ciphertext stays where it is, in the log and in the backups, and becomes permanently unreadable — which regulators have generally accepted as putting the data beyond use. The event log keeps its ordering and its hash chain, projections keep their shape, and the personal content is gone.

Backups need the second mechanism as well, because a restore would otherwise resurrect a deleted record. A suppression list — subject identifiers with erasure dates — is applied as part of the documented restore procedure, so recovery cannot silently undo compliance. The procedure has to be written down and tested, because it is the control an auditor will ask about.

The detail candidates leave out is the derived copies. Deleting from the primary database while the search index, the analytics warehouse, the support tool and last week's CSV export still hold the record is not erasure, and each of those copies was created by a normal engineering decision. That is why erasure is easier to design for at the start than to add later, and why a registry of every store holding personal data is the actual deliverable.

A data protection impact assessment is a structured risk assessment required before processing likely to result in high risk to individuals — large-scale profiling, systematic monitoring, special-category data at scale, automated decisions with legal effect, or new technology applied to personal data.

1  Description of processing
   what data, whose, how much, how long, where it flows, which sub-processors
2  Necessity and proportionality
   the purpose, the lawful basis, why less data or a less intrusive method
   will not achieve it, how minimisation is applied
3  Consultation
   who was asked - the DPO, security, affected users or their representatives
4  Risks to individuals
   each risk framed as harm to a person, not to the company:
   discrimination, financial loss, identity theft, loss of confidentiality,
   inability to exercise rights, distress
5  Measures to address each risk
   the technical control, the residual risk, and who accepted it
6  Sign-off and review
   DPO opinion, decision to proceed, and the trigger for revisiting

Section four is the one engineers get wrong, and it is the section that defines the exercise. A DPIA assesses risk to the data subject, not risk to the business, so "reputational damage" is not a DPIA risk while "a user is wrongly declined credit and cannot find out why" is. Reframing every risk as a harm to a person changes which mitigations look adequate.

Section two is where architecture is decided. Proving that you cannot achieve the purpose with less data forces the design conversation early: could this run on aggregates, could the identifier be pseudonymised at ingest, could the retention be ninety days instead of indefinite. A DPIA done after the build is a justification exercise; done before, it is a requirements document.

The practical reason to care in an interview is that a DPIA is the artefact that gets a feature involving personal data approved. Being the engineer who can draft one, rather than waiting for legal to produce it, is the difference between a two-week block and a conversation.

As an append-only record of events, not a mutable flag. Each consent event needs the subject, the specific purpose, the timestamp, the version of the wording shown, the mechanism of capture, and whether it was granted or withdrawn — because the obligation is to demonstrate that consent was validly obtained, and a boolean demonstrates nothing. Withdrawal must be as easy as granting, must take effect across every downstream system rather than only the one that owns the flag, and does not retroactively invalidate processing already performed lawfully. The common failure is a marketing platform that keeps sending because it holds its own copy of the flag, which is a genuine breach caused by an ordinary integration.

That the data physically stays in permitted jurisdictions, which turns a deployment detail into a design constraint. In practice it means region-pinned storage with a routing layer that knows which region owns each tenant, backups and disaster-recovery copies in permitted regions only, and care with everything peripheral: logs, metrics, error traces, search indexes, queues, third-party SDKs and support tooling all copy data somewhere. Cross-border transfer needs a legal mechanism such as standard contractual clauses plus a transfer risk assessment. The consequences are operational too — a global support team may not be able to view a record, and a single global aggregate view may have to be built from in-region aggregates rather than raw rows.

GDPR gives seventy-two hours from becoming aware of a personal data breach to notify the supervisory authority, and other regimes have their own clocks. Since the notification must describe the categories and approximate number of records affected, the clock is really a demand for forensic readiness: you need logs that show which records were accessed, retained long enough to look back, with integrity you can defend. Without that, the honest answer to "what was taken" is "we cannot tell", which regulators treat far worse than a bounded loss. So the engineering deliverables are access logging on sensitive data paths, sufficient retention, and a rehearsed process — because determining scope is the slow part, not writing the letter.

Five: security, availability, processing integrity, confidentiality and privacy. Security is mandatory and the others are included in scope by choice, which is the first thing to check about any report you are shown — a SOC 2 covering security alone says nothing about uptime. The criteria are objectives rather than requirements, so unlike a prescriptive standard there is no fixed control list; you define controls against each criterion and the auditor tests the ones you described. That flexibility is why two SOC 2 reports can describe very different security postures, and why reading the controls and the exceptions matters more than the fact of the report.

Population and evidence, repeatedly. Give me every production change in the period, then let me sample fifteen and show me the approval for each. Give me every joiner and leaver, then show me the access grant and the revocation dates. Show me the alert that fired for this incident and the ticket that resolved it. Show me the backup restore test. Show me the access review with the reviewer's name. The pattern is population first, sample second, so the awkward question is not "do you review changes" but "prove this list is complete". Systems that can export a complete population make an audit routine; systems that cannot make it an archaeology project.

Type I attests that controls were suitably designed at a point in time. Type II attests that they operated effectively over a period, typically three to twelve months. That distinction is the whole value: a Type I says you had the right policies on one day, a Type II says the control actually ran every time for a year, which is why customers ask for Type II and why the observation window matters. For engineering it means evidence must be continuous rather than assembled for the assessment, and a control introduced halfway through the period gets tested only from its start date — so gaps in the middle of the window are visible in the report as exceptions.

The document listing every control from Annex A, whether you have applied it, and the justification either way. It is the centre of the standard because ISO 27001 certifies a management system rather than a fixed control set: you are allowed to exclude a control, but you must say why, and the auditor tests you against what you claimed. The engineering relevance is that the statement of applicability is where "we do not do X because it does not apply to us" becomes a formal position somebody signed. It also means the interesting question about a certified vendor is not whether they hold the certificate but what their scope and exclusions are.

Everything that stores, processes or transmits cardholder data is in scope, and scope is the entire game — the requirements are demanding, so the winning move is to reduce what they apply to.

BEFORE - the application handles the card

  browser --> our web tier --> our API --> our database
                                            stores PAN, expiry, cardholder name

  in scope: web tier, API, database, backups, logs, the network segment they
            sit in, every admin workstation with access, the CI system that
            deploys them, the developers, the log aggregator
  obligations: quarterly scans, penetration testing, network segmentation,
            key management, full audit logging, annual assessment of all of it

AFTER - tokenisation at the boundary

  browser --> payment provider iframe --> provider vaults the PAN
          <-- token pm_9f2a returned
  our API --> stores pm_9f2a only, uses it to charge later

  in scope: the page that embeds the iframe, and the integrity of that page
  out of scope: our database, our backups, our logs, our warehouse, our
            support tooling - none of them ever see a PAN
  obligations: a far shorter self-assessment questionnaire

The mechanism is worth stating precisely: the token is useless to an attacker because it has no mathematical relationship to the card number and can only be exchanged for a charge by an authenticated call from us. So a dump of our database is not a cardholder data breach, which is a different conversation with a regulator and with customers.

What tokenisation does not remove is the boundary itself. The page hosting the iframe stays in scope, because a compromised script on it can capture keystrokes before the provider ever sees them — which is exactly the attack that motivated the newer requirements on script integrity. "The provider handles cards, so we are out of scope" is the overconfident version of this answer.

The transferable principle beyond payments is that the cheapest way to satisfy a control is to remove the data it protects. The same reasoning applies to national identifiers, health data and credentials: hold a reference, keep the sensitive value in a system built to hold it, and the audit surface shrinks to the boundary. That framing — scope reduction before control implementation — is what an interviewer is listening for.

It governs protected health information — health data tied to an identifiable person — and reaches engineering through three rules. The privacy rule limits use and disclosure to the minimum necessary, which pushes toward field-level access control rather than role-wide access to a record. The security rule requires administrative, physical and technical safeguards, of which the engineering ones are access control, audit controls, integrity, authentication and transmission security. The breach notification rule sets reporting duties. The practical consequences are a business associate agreement with every vendor touching the data, access logging detailed enough to show who viewed which patient record, and encryption that acts as a safe harbour reducing notification obligations.

That the product is secure today. A certificate covers a defined scope during a defined period, assessed against controls the organisation largely chose, and it says nothing about the systems excluded from scope, the vulnerabilities nobody wrote a control for, or what changed the day after the assessment ended. It is evidence of process maturity rather than of resistance to attack. The sophisticated way to use one in a vendor conversation is to read the scope, the list of controls and the exceptions, and to ask what is outside the boundary — which is where the interesting answers are, and which the certificate itself never advertises.

Sarbanes-Oxley makes company officers personally accountable for the accuracy of financial reporting, and it reaches engineering through the controls over any system that feeds the financial statements. In practice that means changes to in-scope systems need documented authorisation, evidence of testing, approval by somebody other than the developer, and a record linking the deployed artefact to the approved change. Developers must not be able to alter production data directly, because that would let a single person change reported numbers without trace. None of this forbids automation — a pipeline that enforces review, records approvals and refuses unreviewed deployment is a stronger control than a human gate.

Know your customer is identity verification at onboarding — document checks, liveness, sanctions and politically-exposed-person screening — producing an auditable record of what was verified, when, and by which provider. Anti-money laundering is ongoing transaction monitoring against rules and models that raise alerts for human investigation. The engineering shape is a case-management system: alerts, investigator queues, decisions with rationale, escalation to a suspicious activity report, and full history because a regulator may review a decision years later. Two constraints define the work — you may not tip off the subject, and false positives dominate volume, so the interesting problem is precision without missing the true cases.

Under PSD2, most electronic payments in Europe need two independent factors from knowledge, possession and inherence, which changes the shape of the checkout. The flow becomes multi-step and interruptible: you initiate, the issuer may challenge, the customer completes a challenge out of band, and you resume — so payment state must be persisted and resumable rather than held in a request. Exemptions exist for low value, recurring amounts, and merchant-initiated transactions, and using them correctly is a real engineering feature because each exemption shifts liability. Systems built on the assumption that authorisation is synchronous and immediate need reworking rather than patching.

It requires firms to record communications that relate or may relate to a transaction, including telephone calls and electronic messages, and to retain them for five years in a form that cannot be altered. That is why regulated trading firms archive email, chat and voice, and why using an unapproved messaging app for business is a serious matter rather than a policy nicety. The engineering consequences are capture at source for every approved channel, immutable storage with retention enforcement, and search good enough to produce a specific conversation on request. It also means new communication tools cannot be adopted until capture exists, which is a common source of friction.

The Digital Operational Resilience Act sets requirements for financial entities in the EU on ICT risk, incident reporting, resilience testing and oversight of third-party providers. Its engineering substance is that resilience must be evidenced rather than asserted: you identify critical business services, set impact tolerances for each — the maximum tolerable disruption — and then test whether you actually stay inside them under severe but plausible scenarios. That reframes disaster recovery from an infrastructure capability into a per-service statement with a number attached, and it makes scenario testing, including losing a major cloud provider, an obligation instead of an exercise nobody funds.

Because a supervisor's question is not whether your cloud provider is reliable but what happens to your critical service if it becomes unavailable or you must leave. Concentration risk is the exposure created by many critical functions depending on one provider, or by an industry depending on the same few. An exit plan is the documented, costed and ideally tested route to another provider or to in-house operation within a stated timescale. The engineering consequence is pressure toward portable abstractions, data export in usable formats, and honesty about which managed services are effectively irreversible — which is a legitimate architectural trade rather than a reason to avoid managed services.

An application log exists for engineers to debug, is verbose, is sampled or truncated when volume rises, and is usually deleted after a few weeks. An audit log exists to answer "who did what to which record, when, and from where" for somebody who may be adversarial, so it must be complete, append-only, retained for years, and protected from the people whose actions it records. Treating them as one system fails both: debug logs are too noisy and too short-lived to be evidence, and audit records are too important to be dropped when a disk fills. The separation is the design point, and the audit sink should be one an application role can write to but not modify.

The fields are not arbitrary; each one answers a question somebody will ask during an investigation.

{
  "event_id":      "01J8Z7QK2P4RXM9V",   // unique, so duplicates are detectable
  "occurred_at":   "2026-07-14T09:22:41.118Z", // UTC instant, ordering evidence
  "recorded_at":   "2026-07-14T09:22:41.204Z", // gap reveals delayed ingestion
  "actor_id":      "u_4821",             // who - a person, never a shared account
  "actor_type":    "user",               // user, service, or system job
  "on_behalf_of":  "u_9902",             // impersonation, or support acting for
  "action":        "customer.pii.viewed",// verb from a closed vocabulary
  "object_type":   "customer",
  "object_id":     "c_77310",            // exactly which record, not "a customer"
  "outcome":       "allowed",            // denials matter as much as successes
  "reason":        "ticket SUP-4471",    // business justification for the access
  "source_ip":     "203.0.113.44",
  "user_agent":    "internal-admin/2.9",
  "session_id":    "s_ab19",             // ties a burst of events together
  "request_id":    "r_5f3c",             // joins to application logs
  "before":        null,                 // prior values on a change
  "after":         null,
  "prev_hash":     "9c1f...",            // chain link, tamper evidence
  "hash":          "42ab..."
}

Four fields carry more weight than the rest. actor_id must resolve to a human or a specific workload — a record attributed to admin or to a shared service account cannot answer the only question that matters, and shared credentials are therefore an audit defect rather than merely an access one. outcome must include denials, because a pattern of refused attempts is how you detect probing, and a log of successes only tells you nothing about the attacker who failed nine times.

reason is the field engineers omit and auditors want. Access to sensitive data by staff is legitimate only in context, so recording the ticket or case that justified it converts an unanswerable question — why did this employee read this customer's record — into a check anybody can run. Making it mandatory at the point of access is far cheaper than reconstructing intent afterwards.

The before/after pair is what makes the log an audit trail rather than an access log. Without prior values you know a field changed and cannot say what it was, which fails both investigation and rectification requests. Store them selectively, since they may themselves contain personal data subject to retention rules.

The hash chain is covered separately, but note the two timestamps: distinguishing when something happened from when it was recorded is what lets you detect a collector that was down, and a single timestamp hides exactly that gap.

Not immutability alone, since anyone able to write can usually also delete. The standard construction is a hash chain: each record includes the hash of its predecessor, so altering or removing an entry breaks every subsequent hash and the break is detectable by recomputation. Periodically publishing the current head hash somewhere outside the system — a separate account, a notarisation service, a signed daily digest — removes the remaining attack of rewriting the whole chain. Combine that with write-only permissions for the application and a separate account holding the store, so the people whose actions are recorded cannot reach it. Detection rather than prevention is the achievable property.

Write once, read many: storage that physically or logically refuses modification and deletion until a retention period expires, implemented in cloud object stores as an object lock in compliance mode. Compliance mode is the important variant, because it prevents even the account root from shortening the retention — which is the point, since the threat model includes a privileged insider and ransomware that deletes backups. It is required outright by some regimes, notably broker-dealer record-keeping rules and several financial archiving obligations, and it is increasingly used voluntarily for backups. The cost is that a mistake is permanent: data written under a seven-year lock cannot be removed for seven years.

Retention is where privacy and record-keeping pull in opposite directions, so the schedule is the artefact that resolves them per category rather than per system.

category                     retain    driver                     deletion mechanism
---------------------------  --------  -------------------------  ---------------------------
Invoices and tax records     7 years   tax law - overrides        archive to WORM, expire
                                       erasure requests           by object lock

Payment authorisations       13 months chargeback window          scheduled purge job,
                                                                  token retained separately

KYC verification evidence    5 years   AML record-keeping         WORM, no early deletion
                             after     obligation
                             closure

Customer account and profile until     contract, then no basis    cascade delete plus
                             closure   to keep                    warehouse purge
                             + 90 days

Support tickets              3 years   legitimate interest,       delete, with PII redaction
                                       dispute defence            at 12 months

Marketing consent records    until     must evidence consent      append-only, survives the
                             withdrawn was validly obtained       marketing data itself
                             + 2 years

Application and access logs  90 days   incident investigation     lifecycle rule on the bucket
Audit trail                  7 years   evidence of controls       WORM, hash chained
CCTV                         30 days   proportionality            automatic overwrite
Backups                      35 days   recovery objective         lifecycle expiry, plus a
                                                                  restore suppression list

The line that surprises people is that consent records outlive the data they authorise. If you delete somebody's marketing profile you still need to show that the earlier sends were lawful, so the consent event is retained on its own basis. The same logic applies to the audit trail: it is the evidence that erasure happened, so it survives the erasure.

The second reading is that retention is not one number. Every row has a different driver and therefore a different owner, and a system-wide "keep everything for a year" policy is simultaneously too long for logs containing IP addresses and far too short for tax records. The schedule is the mapping from legal driver to mechanism, and the mechanism column is what makes it real — a schedule with no enforcing job is a document, not a control.

Third, the conflicts are decided in advance rather than per request. A tax obligation beats an erasure request for the invoice and does not extend to the marketing profile, so an erasure request produces partial deletion with a recorded exemption. Deciding that during a one-month statutory clock, per subject, is how teams miss the deadline.

The engineering deliverable is that every store has a category, every category has an expiry mechanism that runs and reports, and deletion is verified rather than scheduled. Auditors ask for proof that data past its retention is actually gone, and a lifecycle rule nobody has checked is a common finding.

The inventory GDPR requires: for each processing activity, its purpose, the categories of data and subjects, recipients and sub-processors, transfers outside the region and their safeguards, the retention period, and the security measures applied. It sounds like a legal register and is really the data map that makes everything else answerable — you cannot honour a subject access request, enforce retention or scope a breach without knowing where personal data lives and why. Kept as a document it is stale within a quarter; the version that works is generated from something engineers already maintain, such as a service catalogue where every datastore declares its categories and purposes.

Classification is only useful when each class implies concrete engineering obligations rather than a label on a document.

class          examples                    storage            access
-------------  --------------------------  -----------------  -------------------------
Public         marketing site, docs,       any                unrestricted
               published API reference

Internal       runbooks, architecture,     internal systems   all employees
               non-personal metrics        only

Confidential   customer PII, contracts,    encrypted, in      role-based, least
               source code, salaries       approved regions   privilege, access logged

Restricted     card data, health records,  encrypted with     named individuals only,
               credentials, KYC evidence   per-tenant keys,   justification required,
                                           WORM where         approval per access,
                                           mandated           full audit trail

class          transmission        retention           in a non-prod environment
-------------  ------------------  ------------------  --------------------------
Public         any                 indefinite          yes
Internal       TLS                 as useful           yes
Confidential   TLS, no third       per schedule        only if pseudonymised
               party without DPA
Restricted     TLS plus payload    per schedule,       never - synthetic data only
               encryption          enforced by job

The bottom-right cell is the one that changes daily engineering behaviour. Copying production data into a test environment is the single most common way regulated data escapes its controls, because the test environment has weaker access control, no audit logging, wider network reach and a longer list of people with credentials. Once classification exists, "restore production into staging to reproduce the bug" is a policy violation with a named alternative rather than a judgement call.

The second useful row is transmission for the restricted class. Payload encryption on top of TLS matters because TLS terminates at your load balancer, and everything downstream — the mesh, the queue, the log aggregator, the error tracker — sees plaintext. Field-level encryption keeps the data opaque to systems that never needed to read it, which shrinks the number of components in scope.

Third, the number of classes should be small. Four is common and workable; teams that define seven find nobody can remember them, so everything defaults to the middle class and the scheme stops distinguishing anything. Fewer classes with genuinely different handling beats a fine-grained taxonomy nobody applies.

The engineering test of whether classification is real is mechanical enforcement. If a restricted field can be logged, exported to a spreadsheet or copied to a test database without anything stopping it, the table is aspirational — so the useful work is tagging fields in code, blocking them at the log serialiser, and failing a build when a new column has no classification.

Every identity holds the minimum permissions needed for its function, and only for as long as needed. The honest implementation differs from the aspiration in two ways. Permissions accumulate, because they are granted for an incident and never removed, so least privilege requires expiry and periodic review rather than careful initial grants. And standing access is the thing to attack: the strong pattern is just-in-time elevation, where an engineer holds read-only access normally and requests time-boxed elevated access with a reason. It applies to service accounts too, which are usually worse offenders than humans because nobody reviews them and their credentials never rotate.

Segregation of duties splits a sensitive process so no single person can complete it alone — the person who requests a payment cannot approve it, the person who writes code cannot release it to production, the person who creates a vendor cannot pay one. A toxic combination is a set of permissions that, held together, defeats the split, and the point is that each permission is individually reasonable. Create-vendor plus approve-payment is fraud with no collusion needed. That is why access reviews look at combinations rather than individual grants, and why the analysis is genuinely an engineering problem: enumerating effective permissions across systems and roles is a graph query, not a spreadsheet.

The control is only real if the system enforces it, because a policy that relies on people declining permissions they hold is not a control.

flowchart LR
    A[Analyst prepares payment batch] --> V[System validates limits<br/>and beneficiary allowlist]
    V --> P1[Approver one reviews<br/>and confirms]
    P1 --> P2[Approver two releases<br/>different person, different role]
    P2 --> G[Gateway executes]
    G --> R[Audit record: preparer,<br/>both approvers, amount, timestamps]
    A -.system blocks.-> P2

The dotted edge is the entire control. The preparer must be technically incapable of approving, which means the check is a server-side authorisation rule comparing the approver's identity to the preparer's, not a hidden button. Client-side enforcement of segregation of duties is a finding every time, because the API is the boundary that matters.

Three loopholes an auditor will probe. Can the preparer edit the batch after the first approval, so approval applies to something other than what executes? The answer must be that any modification voids approvals and restarts the chain. Can one person hold both approver roles by virtue of a broad group membership? That is the toxic combination, and it is found by evaluating effective permissions rather than assigned roles. And can somebody with database access alter the beneficiary after release, which is why the account holding the payment table must not be reachable by an interactive human session.

The delegation case is where real implementations leak. Approvers go on holiday, so delegation exists, and a poorly designed delegation lets the preparer be somebody's delegate. Delegation must be time-boxed, recorded as an event, and subject to the same identity comparison as direct approval.

The transferable point is that the pattern generalises well beyond payments — code to production, refunds, privilege grants, customer data exports. In each case ask who can complete the loop alone, then make the system refuse. If the answer is "nobody would", the control does not exist yet.

The set of controls around accounts that can change or destroy things: a vault holding the credentials so humans never see them, checkout with an approval and a reason, short lifetimes, session recording for interactive access, and automatic rotation on release. The reason it is treated separately from ordinary access control is that privileged accounts are the target — one of them is the whole compromise — and they are also the accounts most likely to be shared, static and unattributed. A well-implemented system means no engineer holds a standing production administrator credential, which removes the most valuable thing an attacker could phish from them.

Emergency access has to exist, because a control that prevents recovery during an outage will be bypassed. The design goal is to make it usable, unattractive and loud.

sequenceDiagram
    participant E as On-call engineer
    participant P as Access broker
    participant S as Production
    participant A as Audit sink
    E->>P: Request break-glass role with incident id
    P->>A: Record request, reason, second approver
    P-->>E: Issue credential valid 60 minutes
    E->>S: Act, with session recording enabled
    S->>A: Stream every command and its output
    P->>A: Revoke on expiry and log the revocation

What the procedure records is the part that makes it acceptable to an auditor:

who              named individual, never a shared break-glass account
when             request, grant, first use, last use, revocation
why              incident identifier, linked to a real incident record
who approved     a second person, or a documented no-approver path with
                 mandatory after-the-fact review within 24 hours
scope granted    which role, which systems, which environment
what was done    full session recording or command log, streamed off the host
closed out       review by someone other than the user, within a fixed window,
                 recorded as complete

The two design decisions worth defending are the time box and the after-the-fact review. A sixty-minute credential means a forgotten elevation cannot become standing access, which is how privilege accumulates in practice. And because a genuine emergency may not allow waiting for an approver, the honest control is mandatory review afterwards rather than pretending pre-approval is always available — every use is examined by a second person whether or not anyone approved it in advance.

The failure mode to name is routine use. If break-glass is invoked forty times a month, it is not emergency access, it is the access model, and the correct fix is to grant the missing routine permission properly rather than to keep logging exceptions. Counting invocations per month is therefore a control health metric, and a rising count is a design signal.

The alerting detail candidates miss is that a break-glass grant should page somebody independent immediately. Its legitimate uses are rare enough that a notification is not noise, and an attacker's use of it is precisely the event you most want a human to see within a minute.

The requirement that two people concur before a sensitive action takes effect, implemented as an authorisation rule rather than a convention. It fails in three recognisable ways. It becomes a rubber stamp when the second person has neither context nor time, which is why approving a five-thousand-line change is nominally compliant and substantively worthless. It fails when both people report to whoever benefits, since independence is the actual requirement rather than headcount. And it fails through delegation and shared accounts, where two identities turn out to be one person. The engineering countermeasures are small changes, an independent approver pool, and identity checks the system enforces.

That access is created, changed and removed from an authoritative source rather than by request. The joiner case is easy; the two that generate findings are the others. A mover keeps their old permissions alongside their new ones unless something removes them, which is how an engineer who moved to finance ends up with a toxic combination. A leaver must lose access everywhere within the promised window, including systems outside single sign-on, personal access tokens, SSH keys, API credentials and third-party tools bought on a card. The engineering requirements are provisioning driven by the HR system, no local accounts, and a reconciliation that finds what the process missed.

Because a change nobody authorised, tested or recorded defeats every control built on top of it — you cannot claim the reported numbers are accurate, or that access is restricted, if one person can alter production at will. The prohibition is not about ceremony but about a single individual having an unreviewed, unlogged path to the data. That rules out interactive database sessions on production, hotfixes applied by hand, and configuration edited on a host. The alternative is not slower delivery: it is that every change travels the same recorded path, so the evidence exists as a by-product and the emergency case is a documented variant rather than an exception.

The record is what an auditor samples, so it is worth knowing exactly what a complete one contains — and how much of it a good pipeline emits automatically.

Change            CHG-8842
Title             Add index on payments.created_at to fix report timeout
Type              Standard - pre-approved category
Requested by      a.patel        2026-07-09 10:14
Risk assessment   Medium. Online index build, no lock. Rollback: drop index.
Systems affected  payments-api, reporting - both SOX in scope

Peer review       PR #4471, approved by r.osei 2026-07-09 11:02
                  author cannot approve - enforced by branch protection
Test evidence     CI run 22193: 412 unit, 38 integration, all passed
                  migration applied and rolled back in staging - log attached
                  query plan before and after, attached
Security review   not required - no change to data flow or access
CAB decision      Standard change, pre-approved template CT-12. No CAB needed.
Release approval  m.lindqvist, release manager, 2026-07-09 14:30
                  separate from author and reviewer

Deployment        pipeline run 8891, artefact sha256:4c1f..., 2026-07-09 15:02
                  deployed by the pipeline identity, no human production access
Verification      report p95 3.2s down from 41s, error rate unchanged
                  post-deploy checks green at 15:20
Closure           closed by m.lindqvist 2026-07-09 16:00

The structural point is the separation across three distinct identities: author, reviewer, releaser. That is segregation of duties applied to code, and it is why "the developer deployed their own fix" is a finding regardless of how good the fix was. The pipeline identity performing the deployment matters as much, because it means no human held production credentials at any point.

The pre-approved standard change is the mechanism that makes this compatible with frequent delivery, and candidates rarely mention it. Rather than reviewing every change in a board meeting, you get a category approved once — low-risk, tested, reversible, matching a template — and changes in that category flow through automatically with the evidence recorded. The board's attention then goes to the changes that actually carry risk. A team deploying twenty times a day under SOX is using this, not skipping change management.

Everything from peer review to verification is emitted by systems that already have the facts. The branch protection rule enforces the author check, CI produces the test record, the pipeline records the artefact hash and the deployer, the monitoring records the verification. What makes an audit painful is a change record typed by a human after the fact, describing what they remember doing.

The field to notice is the artefact hash. Linking the approved change to the exact binary deployed is what closes the gap between "this change was approved" and "this code is running", and without it an auditor can reasonably ask how you know they are the same thing.

An artefact tying a specific test execution to the specific change, retained for the audit period — a pipeline run identifier with results, coverage of the affected area, and the record that it passed before deployment. What does not count is a policy stating that changes are tested, a green tick nobody can retrieve six months later, or a developer's assertion. The consequence for engineering is that CI becomes a system of record: its retention matters, its results must be attributable to a commit and an artefact, and deleting old runs to save storage destroys evidence. Manual testing counts too, provided somebody recorded what was executed, by whom, and with what result.

A change that bypasses the normal approval sequence because the alternative is prolonged harm — a production outage, an active security incident. It stays legitimate through three things: a defined trigger so it cannot be used for convenience, the same technical path as a normal change so the artefact and the deployment are still recorded, and a mandatory retrospective approval and review within a fixed window. Auditors look at the count and the distribution: a handful a year is credible, several a week means the normal process does not work and the real finding is that the organisation has two processes, one of which has no controls.

Yes, and it is usually a stronger control environment than a manual one, provided the pipeline enforces what a human gate used to. Peer review enforced by branch protection, tests as a hard gate, provenance recorded per artefact, deployment performed by an identity no human can borrow, automatic rollback, and standard changes pre-approved as a category rather than individually. The argument to make is that a machine applies the control every time and produces the evidence automatically, whereas a change advisory board approves from a description and leaves a minute. What genuinely does not survive is a developer with production credentials and a deploy script.

Because a recovery capability nobody has exercised is an assumption, and several regimes require evidence rather than a plan — a documented test with a date, an operator, the scenario, the measured recovery time and point, and the defects found. The engineering value is that these tests always find something: expired credentials in the recovery path, a dependency in the failed region, a runbook referring to a decommissioned host, a backup that restores but whose application cannot start. Restoring backups regularly is the minimum; the version that satisfies an operational resilience regime is failing a critical service and measuring whether you stayed inside its stated impact tolerance.

Encryption is claimed as a control far more often than it is reasoned about, and the discipline is to say what each layer protects against.

layer         mechanism                 protects against              does NOT protect against
------------  ------------------------  ----------------------------  --------------------------
At rest       full-disk or volume       stolen disk, decommissioned   anything with a valid
              encryption                hardware, raw storage         database session - the
                                        access by the platform        application sees plaintext

At rest       transparent database      the same, plus file-level     a compromised application
              encryption                access to the data files      account, a SQL injection,
                                                                      a privileged DBA

At rest       application field-level   the database, the backups,    the application itself,
              encryption                the replicas, the logs, the   and anyone holding the key
                                        warehouse, the DBA

In transit    TLS on external hops      network interception,         everything past TLS
                                        tampering, some spoofing      termination - mesh, queue,
                                                                      logs, error tracker

In transit    mutual TLS internally     a workload impersonating      an authorised workload
                                        another                       misusing its access

In use        confidential computing    the host operating system     application logic flaws,
              or enclaves               and cloud operator            and side channels

The row that reframes most discussions is application field-level encryption, because it is the only one that reduces audit scope. Volume encryption protects against a threat — someone walking out with a disk — that is largely already handled by a cloud provider, and it does nothing about the compromise you actually expect, which is an attacker with valid application credentials. Encrypting the sensitive field so that the database, the replicas, the backups and the analytics warehouse hold only ciphertext genuinely removes those systems from scope.

The in-transit rows deserve the same scepticism. "All traffic is encrypted" usually means TLS terminates at the edge, after which the payload is plaintext through the service mesh, the message broker, the log pipeline and the error tracker — and it is the error tracker holding a full request body that turns up in incident reports. Deciding which hops need encryption is a design question, not a checkbox.

The cost column that is missing from the table is queryability. Encrypted fields cannot be indexed, range-queried or joined without deterministic or order-preserving schemes that leak information, so field-level encryption forces choices about which fields need lookup. That trade — precision of protection against ability to query — is the substance of the answer, and it is why the decision is made per field rather than per system.

Because the algorithm is never the weak point and the key handling always is. If the key sits next to the data, in an environment variable, in a repository, or in a vault every service can read, then encryption protects against physical theft and nothing else. The control is the boundary around the key: a managed service or hardware module where the private key cannot be exported, authorisation policies saying which identity may use which key for what operation, separation so the people who administer keys are not the people who administer data, audit logging of every use, and rotation with the ability to re-wrap. Crypto-shredding depends entirely on this, since deleting a key is only a deletion if no copy survives in a backup.

Nothing verifiable, which is why the phrase is a warning rather than an assurance. There is no GDPR certificate to hold; compliance is a continuous property of how you handle data, and the only meaningful statements are specific ones — we have a lawful basis recorded per purpose, we can fulfil an access request in eleven days, we have tested erasure across nine stores, our register of processing is generated from the service catalogue. In an interview the trap is answering at the level of the slogan. Naming the mechanism and the last time it was exercised is what distinguishes somebody who has done the work from somebody who has read the summary.

No, and proposing it is the wrong instinct, because the audit log is itself the evidence that your controls operated and that the erasure happened. Two things resolve the apparent conflict. First, the audit record should not contain personal data beyond identifiers in the first place — it records that a subject's record was viewed, not the contents. Second, where personal data is unavoidable, encrypt it per subject and destroy the key, so the entry, its ordering and its hash chain survive while the content becomes unreadable. Legal record-keeping obligations also lawfully override erasure, and saying so, with the exemption recorded, is the correct answer.

Only if you do not hold the key. Encryption at rest with a key your application can use on demand keeps the data fully in scope, because the compromise you are worried about — valid credentials, injection, a rogue insider — happens on the side of the boundary where plaintext exists. Scope reduction comes from the data not being present in a recoverable form: tokenisation where the mapping lives with a provider, or field-level encryption where the key is held by a system a different team controls. The distinction interviewers listen for is between encryption as a protection and encryption as a scope boundary, and only the second requires the key to be somewhere you cannot reach.

Three things at once. Logs containing personal data are themselves in scope, so verbose logging creates copies of regulated data in a system with weak access control, wide readership and no subject-access story — and it is the most common way PII escapes its classification. It conflates debugging with evidence, so the records that matter are diluted by noise and dropped when volume rises. And it makes retention undefined, because nobody wrote a policy for the log bucket, which turns a data-breach question into an unbounded one. The mature answer is a deliberate split: minimal, scrubbed application logs with short retention, and a separate audit trail designed as evidence.

Whether you can turn a compliance requirement into an engineering design, without either dismissing it or reciting regulation. A strong answer identifies the questions the system must be able to answer — who accessed this record, what changed and to what, who approved this release, is this data past its retention — and then designs backwards: a separate append-only audit store with a hash chain, mandatory actor and justification on sensitive reads, before-and-after values on change events, evidence emitted by the pipeline rather than assembled by a human, retention enforced by a job that reports, and a rehearsal proving the questions can be answered. It names the costs too, because audit logging adds write volume, retains regulated data and constrains schema changes.

A weak answer substitutes vocabulary for design. It says the system will be compliant, lists regulations by name, promises to log everything, or treats the problem as a policy somebody else owns. The distinguishing signal is whether the candidate has ever had to produce evidence for a period that had already passed — because that experience is what teaches you that auditability is a property you build in advance or not at all.