Regulation & Compliance

What this area actually covers

This is the study of rules that constrain how a system may be built and operated, and of how you prove to an outsider that you followed them. It crosses every other domain here: a payments ledger, a hospital record, a telecoms bill and a factory historian share almost no technology, yet all four get asked where the data physically lives, who may read it, how long it is kept, and who approved the last change to production. The subject matter is residency and cross-border transfer, the obligations privacy law places on a system, auditability and tamper-evident evidence, access control with separation of duties, and how each of those is evidenced rather than asserted.

The boundary that matters is with security engineering. Security reduces risk; compliance demonstrates, to someone who does not trust you and cannot read your code, that you reduced it the way you said you would. A well-secured system can fail an audit because nothing was recorded, and a weak one can pass because the audit examined only the scope the company itself defined. Two things get wrongly bundled in: legal interpretation, which belongs to a lawyer or a data protection officer rather than to you, and contractual regimes, which are not law even when they behave like one — PCI DSS comes from the card brands through the PCI Security Standards Council, and a customer's security addendum can bind you more tightly than a statute.

A third thing gets wrongly excluded, which is the operational half. Most engineers picture compliance as a design-time constraint and stop there, but the majority of findings in real audits are not architectural. They are a leaver whose access was never revoked, a quarterly review that was skipped in August, an alert that fired into a channel nobody read, a vendor onboarded without a review. Compliance engineering is as much about processes that keep running as about systems that are correctly shaped.

The five areas underneath

The section splits along the natural lines of what a regulator, a customer or an auditor actually asks about. Three of the five are about data — what you may do with it, where it may be, and what record you keep of what was done — one is about people and their permissions, and the last is about the schemes under which all of it gets externally examined.

Data Privacy

Privacy law attaches duties to personal data, and those duties are the part that reaches your code. You need a lawful basis for each purpose and you may not quietly reuse the data for a second purpose; you collect no more than the purpose needs; you tell people what you hold and correct it when it is wrong; you delete it in defined circumstances; and you do not keep it indefinitely because it might be useful. Under the GDPR the response to a subject request is time-bound — generally one month, extendable in limited circumstances — which turns a legal duty into an operational one with a service level. This subsection covers lawful basis and consent as distinct things, the subject rights and what each one demands of a system, data minimisation as a design constraint, and the erasure problem in full: the row you delete is also in the backup, the event log, the warehouse, the search index and possibly a training set. Open it for how crypto-shredding, pseudonymisation and bounded retention windows turn an impossible request into a defensible answer.

Data Residency & Transfers

Residency is the requirement that data physically rests in a named place; transfer is the separate question of whether it may cross a border at all, and under what mechanism. They are treated together because they arrive together and because both are cheap at design time and brutal afterwards. This subsection covers what actually changes when residency lands on an existing system: region-pinned deployments, routing that must decide before the write rather than after, identity and control planes that may be global while data planes are not, and the copies people forget are copies. Under the GDPR, moving personal data to a third country needs a lawful transfer mechanism such as an adequacy decision or standard contractual clauses; other regimes state their own rules and some sectors add stricter ones. Open it for the architectural patterns, for the subprocessor question, and for why telemetry rather than the product is usually what creates the transfer nobody declared.

Auditability & Evidence

An audit trail is not a debug log, and this subsection is about the difference. A debug log may be sampled, may be dropped under load, and is written by and for the people who run the system. An evidentiary record must say who did what to which record, when, and what changed, and it must hold up on the assumption that the person being investigated had administrative access — which is why it is append-only, written to storage the application's own operators cannot rewrite, often hash-chained so removing one entry breaks verification, and retained for a period set by obligation rather than by disk cost. Open it for what an auditor accepts as evidence and what they reject, for the difference between an artefact that shows a control was designed and one that shows it operated across a period, and for how to produce a complete population rather than a convincing sample.

Access Control & Separation of Duties

Almost every audit finds something here, and almost none of it is about the access model. The model is usually fine. The lifecycle is where it breaks: accounts that outlive employment, permissions accumulated across three internal moves and never removed, standing production access held by people who need it twice a year, service accounts nobody owns. This subsection covers least privilege in practice, joiner-mover-leaver automation driven from the HR system as the authoritative source, periodic recertification by named owners rather than by IT, just-in-time elevation tied to a ticket, and privileged access management. Separation of duties gets its own treatment because it is a design constraint rather than a policy: no single person should both perform and approve a sensitive action, whoever administers the audit store should not administer the system it records, and the combinations that are individually reasonable but jointly dangerous are called toxic combinations for a reason. Open it for how to express those combinations so they can be tested continuously.

Standards & Certification

This is where most engineers first meet the subject, usually because a deal is blocked on it, and it is the area most confidently misdescribed. This subsection covers SOC 2, ISO/IEC 27001 and PCI DSS in the terms an engineer needs: what each scheme actually examines, who issues the output, what the scope boundary means, and what each one does not say. The single most useful thing in it is negative knowledge — that a report covers a defined scope over a defined period, that "certified" is the wrong word for some of these schemes, and that a passing result is a statement about consistency with commitments rather than a statement that a system is secure. Open it before you answer your first customer security questionnaire.

Where it sits in a real business

An obligation travels a long way before it reaches your code. A regulator writes a rule about an outcome; legal turns it into a policy the organisation commits to; risk and compliance turn that policy into controls, meaning testable assertions about how the organisation behaves; engineering implements each control and, separately, produces the evidence that the implementation ran. An auditor examines the evidence and never the intention.

flowchart TD
  A[Regulator or contract states an outcome] --> B[Legal writes a policy commitment]
  B --> C[Risk defines a control as testable behaviour]
  C --> D[Engineering implements a mechanism]
  D --> E[Evidence is produced continuously]
  E --> F[Auditor samples the population]
  F --> G{Operated all period}
  G -- yes --> H[Clean opinion for that control]
  G -- no --> I[Exception recorded in the report]

The step people skip is the one between the mechanism and the evidence. A mechanism that works but leaves no exportable record produces the same audit outcome as no mechanism at all, which is why the arrow from D to E is drawn separately rather than folded into implementation.

The distinction engineers miss is that a control, its implementation and its evidence are three separate artefacts, and having two of them is worth nothing.

Evidence has to cover a population, not an anecdote: the auditor asks for every production change in the period, then samples it. If you cannot enumerate the population you have no control, however good your review culture is. "We always review pull requests" is a habit; "reviews are enforced by branch protection and exported monthly" is a control.

That reshapes logging too. An audit trail is not a debug log, which may be sampled or dropped under load; an evidentiary record must say who did what to which record, when, and what changed, assuming the person being investigated held administrative access — hence append-only writes to storage the application's own operators cannot rewrite, such as object storage under an object-lock policy, and hash-chained records so that removing one breaks verification. Access control shifts the same way, from model to lifecycle: the recurring findings are leavers' accounts, permissions accumulated through internal moves, and standing production access, answered by joiner-mover-leaver automation from the HR system, recertification by named owners, and just-in-time elevation against a ticket. Separation of duties then keeps any one person from both performing and approving a sensitive action, so the author is not the sole approver and whoever administers the audit store does not administer the system it records.

Commercially, the reason all of this is funded is simpler than the machinery suggests. Compliance sits on the revenue path. An enterprise buyer's procurement process will not let a contract through without a security review, a subprocessor list and usually a current report; a regulated buyer adds an audit-rights clause and a right to be notified of incidents. So the artefacts a compliance function produces are, in commercial terms, inputs to sales, and the deadline that actually moves an engineering team is a renewal rather than a regulator.

The vocabulary of the first fortnight

The terms below are used constantly and precisely by the people who own this work, and loosely by everyone else. Using them precisely yourself is one of the cheapest credibility signals available in an interview or a first week.

Three of these repay attention immediately. Controller and processor decide who owes which duty, and a vendor that quietly behaves as a controller over customer data has created a problem no contract clause will paper over. Compensating control is the phrase that lets an honest engineering constraint be accepted rather than concealed, and it works only if you propose it before the audit rather than during it. And exception is not a failure state: an audit report with no exceptions and a wide scope is rarer than candidates imagine, and recording one properly is the behaviour being tested.

Who does this work

Security and compliance engineers, sometimes titled GRC engineers, build the mechanisms and the evidence pipelines — control mapping, automated configuration checks, evidence exports, the auditor's request list — and spend as much time arguing whether an existing mechanism satisfies a control as writing code. It is a role with an unusual centre of gravity: the deliverable is frequently an argument, supported by an export, rather than a feature.

Privacy engineers own data inventories, retention and deletion machinery and subject-request handling, alongside a data protection officer, a role the GDPR requires of some organisations, who interprets rather than implements. GRC analysts run the control framework itself, map one scheme's requirements onto another's so that a single piece of evidence serves several, and chase owners. Internal auditors test independently and report to the board, not to engineering, which is the point of them; treating internal audit as an adversary is a career-limiting instinct, because they are usually the cheapest way to find out what the external examination will say. Architects in regulated industries carry the subject as background, because their real contribution is noticing at design time that a requirement has created a cross-border transfer.

The split to internalise is between people who specify controls and people who implement them; when nobody does the second, the year ends in a scramble for screenshots.

Demand, adoption and how that is changing

Demand is high for commercial rather than idealistic reasons: compliance gates revenue. Enterprise and public-sector buyers will not sign without a security review, so a vendor without a current report loses deals it never competes for. The number of jurisdictions with comprehensive privacy law keeps growing, and AI systems have added obligations around transparency, data provenance and human oversight that reach engineers as lineage and evaluation work.

It is also where most engineers meet the two regimes, and both are widely misdescribed. A SOC 2 report is an attestation by an independent audit firm under AICPA standards against the Trust Services Criteria; security always applies, while availability, confidentiality, processing integrity and privacy are in scope only if the company chooses. Type 1 covers whether controls were suitably designed at a point in time, Type 2 whether they also operated effectively across a period, which is why buyers want Type 2. Nobody is "SOC 2 certified": the output is a report holding the company's own system description, the auditor's opinion, the tests performed and any exceptions, and it is normal both to have exceptions and to leave products out of scope.

ISO/IEC 27001 is genuinely certifiable, by an accredited body, but it certifies an information security management system — the process by which you assess risk, select controls and improve — rather than any particular technical control, and you are audited against your own risk assessment and statement of applicability. Neither says a system is secure. Both answer a narrower question: does this organisation do what it says, within a scope it defined, and can it prove it.

PCI DSS is different in kind again and worth separating in your head. It is a contractual standard maintained by the PCI Security Standards Council and enforced through the card brands and acquirers rather than by a regulator, it applies to entities that store, process or transmit cardholder data, and the single most consequential engineering fact about it is that scope is reducible. Segmenting the network and keeping card data out of your systems — by tokenising or by letting a payment provider capture the details directly — shrinks what is in scope, which is why so much payment integration is designed to keep the primary account number away from your servers entirely.

Evidence collection itself has commoditised, since compliance platforms and cloud configuration rules now check control state continuously. Demand moved upstream rather than away: the scarce skill is designing systems whose compliant behaviour is a property of the architecture. The platforms will tell you that a bucket is public. No platform will tell you that the feature you shipped last sprint created a transfer of special-category data to a subprocessor in a jurisdiction with no adequacy decision.

What makes it hard

Obligations are written about outcomes, in deliberately technology-neutral language so they outlive implementations, so turning "appropriate technical and organisational measures" into a mechanism is a judgement call. You improve at it by reading how auditors and regulators treated similar arguments, not by reading more statute. There is no lookup table from rule to mechanism, and anyone selling you one is selling a checklist that will be wrong for your architecture.

And nearly everything here is cheap at design time and close to impossible later. Residency, once required, turns a global database into region-pinned deployments and forces routing decisions before the write rather than after, then breaks on the copies you forget are copies: the search index, the warehouse, the logging pipeline, a support tool with production access, any subprocessor. Cross-border transfer is the related question of whether data may leave at all; under the GDPR regimes that needs a lawful mechanism such as an adequacy decision or standard contractual clauses, and the transfer nobody noticed is usually created by telemetry rather than by the product.

Erasure is the sharpest case. Learn privacy regimes at the level of the duties they create — name a lawful basis for each purpose, use the data only for that purpose, collect no more than you need, tell a person what you hold and correct it, delete it in defined circumstances, stop keeping it indefinitely — and the collision is immediate. Deleting the row does not delete the data.

sequenceDiagram
  participant S as Data subject
  participant P as Privacy desk
  participant App as Application
  participant DW as Warehouse
  participant B as Backups
  S->>P: Requests erasure
  P->>App: Verify identity and locate records
  App->>App: Delete row and destroy per subject key
  P->>DW: Propagate to derived copies
  P->>B: Add to suppression list
  B-->>P: Restore replays the list before reuse
  P->>S: Confirms within the statutory window

The interesting participant is the last one. Backups are not deleted from; they are handled by a bounded retention window plus a documented rule that restored data is re-processed against the deletion list, and being able to say that sentence calmly is what distinguishes a credible answer from a fantasy about instant physical erasure everywhere.

The workable answers are architectural. Crypto-shredding puts each subject's data under a per-subject key you can destroy, so ciphertext sitting on immutable media becomes unreadable without being rewritten. Pseudonymisation keeps most systems holding tokens, so only the identity vault has to honour erasure. Tombstoning and compaction handle keyed event logs. And automatic expiry beats deletion on request for the simple reason that it runs without being asked, which is also why a retention schedule is the highest-leverage privacy artefact most organisations do not have. Some copies you are required to keep: an audit trail you can delete from is no longer evidence.

Obligations also conflict in ways that have no technical resolution. A deletion request meets a retention duty; a litigation hold freezes data a policy is trying to expire; a fraud model wants history that data minimisation says you should not keep; an anti-money-laundering duty to retain records sits directly against a customer's request to be forgotten. You resolve those by naming the competing bases, deciding deliberately and writing it down. The written record is the deliverable — an unrecorded good decision and a bad decision look identical eighteen months later.

And evidence is continuous. A control that worked in March and broke silently in July fails a period-based examination even if it is healthy on audit day, so control failure needs alerting exactly like service failure does. The mental model to adopt is that a control has uptime.

stateDiagram-v2
  [*] --> Designed
  Designed --> Operating: mechanism live and exporting
  Operating --> Drifted: config changed or job stopped
  Drifted --> Operating: detected and repaired
  Drifted --> Exception: period ends while broken
  Operating --> Attested: period ends healthy
  Exception --> [*]
  Attested --> [*]

Note that the path to an exception does not require anybody to do anything wrong; it only requires drift to outlive the period. That is the whole argument for continuous control monitoring in one transition.

The last difficulty is scope, and it is the one that produces genuine disagreement rather than genuine difficulty. A narrower scope is easier to pass and less useful to the buyer; a broader one is honest and expensive. Engineers tend to assume scope is a technical fact and are surprised to discover it is a negotiated position, argued between sales, compliance and the audit firm, with real consequences for what you will be asked to build next quarter.

Why study it

Study it if you want to be trusted with architecture in a regulated industry, or you work somewhere that sells to enterprises. The leverage is that very few engineers can look at a design and say "this creates a cross-border transfer and we have no mechanism for it" or "we cannot honour erasure here, because state is rebuilt from an immutable log". Said at the right moment, that saves a rebuild, and unlike a framework it is still true in ten years and in another industry.

There is a career-shaped reason too. Compliance knowledge is one of the few things that makes an engineer legible to executives, because it connects a design decision to a contract, a deal or a fine in one step. That is a useful property if you want to move towards architecture or towards leadership, and it is close to free if you are already in a regulated business, since the obligations are being explained to you anyway.

Skip it if you want depth in algorithms, machine learning research or systems performance, where the overlap is small. Much of the work is written reasoning and negotiated definitions, so if you dislike producing documents other people argue with, you will enjoy the design slice and resent the rest. And be honest with yourself about tolerance for work whose success condition is that nothing happens.

Your first hour

Take one piece of personal data in a system you know — an email address, a date of birth — and trace it. Design nothing. Write down every place it comes to rest and what would happen if its owner demanded erasure today.

Finding four copies you had forgotten is the normal outcome, and the finished table is what a privacy team calls a data inventory. Two columns are doing the real work: "deleted by", which is empty far more often than anyone expects, and "how it got there", which usually names a pipeline built for an unrelated reason by someone who is no longer on the team.

Then take one control — "production changes are approved by someone other than the author" — find the mechanism enforcing it, and try to export last month's complete list of production changes with approvers. Write the export as something repeatable rather than as a one-off query, because repeatability is the whole difference between evidence and an anecdote.

Control    CHG-01  Production changes are approved by a non-author
Period     2026-06-01 to 2026-06-30
Population every merge to the deployable branch in the period
Export     id, author, approver, merged_at, ticket, deployed_at
Checks     approver is never equal to author
           every deploy maps to exactly one merge in the population
           count of merges matches count of deploy records
Exceptions 2 emergency deploys on 14 June, approved retrospectively

Failing to produce that list in an hour is how you discover why this discipline exists — and if you do produce it, the two emergency deploys at the bottom are more instructive than the thirty clean ones, because how you handle a documented, retrospectively approved exception is exactly what an auditor is trying to learn about you.

What this is not

It is not paperwork added at the end, though it is often practised that way, which is where the reputation comes from. Every organisation that treats it that way pays for it in the same currency: a fortnight of engineering time each year spent taking screenshots, and an architecture that has to be unpicked the first time a large customer asks a residency question.

It is not security: security reduces risk, compliance demonstrates you did what you committed to, and either can exist without the other. A system with strong encryption, no logging and no access reviews is secure and unauditable. A system with weak controls, immaculate records and an honest scope statement can pass.

It is not legal advice, and offering it as such is a real risk rather than a stylistic one. The correct engineering posture is to describe the mechanism and its properties precisely and let the lawyer or the data protection officer decide whether those properties discharge the obligation. "Here is what the system does and here is what we can prove about it" is always in your remit; "that satisfies the regulation" usually is not.

And it is not a certificate you collect. A report covers a defined scope over a defined period, which is why answering a security questionnaire by attaching a PDF proves far less than people assume — and why reading someone else's report properly means turning straight to the scope statement and the exceptions rather than the opinion.

A control you cannot evidence for the whole population is not a control, it is a habit, and habits do not survive an audit.