Regulation is a design input, not paperwork bolted on at the end. This is how privacy, residency, auditability and access rules reach into a system, and how a control is evidenced rather than asserted.
High demand22 min readSecurity & compliance engineer, Privacy engineer, GRC analyst, Solution architect, Internal auditor, Data protection officerUpdated 2026-07-27
Assumes you know: Basic system design vocabulary (services, databases, replicas, backups), How authentication and authorisation differ, What a deployment pipeline does
Overview
What this area actually covers
This is the study of rules that constrain how a system may be built and operated,
and of how you prove to an outsider that you followed them. It crosses every
other domain here: a payments ledger, a hospital record, a telecoms bill and a
factory historian share almost no technology, yet all four get asked where the
data physically lives, who may read it, how long it is kept, and who approved the
last change to production. The subject matter is residency and cross-border
transfer, the obligations privacy law places on a system, auditability and
tamper-evident evidence, access control with separation of duties, and how each
of those is evidenced rather than asserted.
The boundary that matters is with security engineering. Security reduces risk;
compliance demonstrates, to someone who does not trust you and cannot read your
code, that you reduced it the way you said you would. A well-secured system can
fail an audit because nothing was recorded, and a weak one can pass because the
audit examined only the scope the company itself defined. Two things get wrongly
bundled in: legal interpretation, which belongs to a lawyer or a data protection
officer rather than to you, and contractual regimes, which are not law even when
they behave like one — PCI DSS comes from the card brands through the PCI
Security Standards Council, and a customer's security addendum can bind you more
tightly than a statute.
A third thing gets wrongly excluded, which is the operational half. Most
engineers picture compliance as a design-time constraint and stop there, but the
majority of findings in real audits are not architectural. They are a leaver
whose access was never revoked, a quarterly review that was skipped in August, an
alert that fired into a channel nobody read, a vendor onboarded without a review.
Compliance engineering is as much about processes that keep running as about
systems that are correctly shaped.
The section splits along the natural lines of what a regulator, a customer or an
auditor actually asks about. Three of the five are about data — what you may do
with it, where it may be, and what record you keep of what was done — one is
about people and their permissions, and the last is about the schemes under which
all of it gets externally examined.
Subsection
What it is for
Data Privacy
The duties attached to personal data, and where they collide with your architecture
Data Residency & Transfers
Where data may physically sit, and what it costs a design when the answer is constrained
Auditability & Evidence
Proving, after the fact, what happened and who did it
Access Control & Separation of Duties
Who may do what, and which combinations of permission are unsafe
Standards & Certification
SOC 2, ISO 27001 and PCI DSS in engineering terms, and what each does not mean
Data Privacy
Privacy law attaches duties to personal data, and those duties are the part that
reaches your code. You need a lawful basis for each purpose and you may not
quietly reuse the data for a second purpose; you collect no more than the purpose
needs; you tell people what you hold and correct it when it is wrong; you delete
it in defined circumstances; and you do not keep it indefinitely because it might
be useful. Under the GDPR the response to a subject request is time-bound —
generally one month, extendable in limited circumstances — which turns a legal
duty into an operational one with a service level. This subsection covers lawful
basis and consent as distinct things, the subject rights and what each one
demands of a system, data minimisation as a design constraint, and the erasure
problem in full: the row you delete is also in the backup, the event log, the
warehouse, the search index and possibly a training set. Open it for how
crypto-shredding, pseudonymisation and bounded retention windows turn an
impossible request into a defensible answer.
Data Residency & Transfers
Residency is the requirement that data physically rests in a named place;
transfer is the separate question of whether it may cross a border at all, and
under what mechanism. They are treated together because they arrive together and
because both are cheap at design time and brutal afterwards. This subsection
covers what actually changes when residency lands on an existing system:
region-pinned deployments, routing that must decide before the write rather than
after, identity and control planes that may be global while data planes are not,
and the copies people forget are copies. Under the GDPR, moving personal data to
a third country needs a lawful transfer mechanism such as an adequacy decision or
standard contractual clauses; other regimes state their own rules and some sectors
add stricter ones. Open it for the architectural patterns, for the subprocessor
question, and for why telemetry rather than the product is usually what creates
the transfer nobody declared.
Auditability & Evidence
An audit trail is not a debug log, and this subsection is about the difference. A
debug log may be sampled, may be dropped under load, and is written by and for
the people who run the system. An evidentiary record must say who did what to
which record, when, and what changed, and it must hold up on the assumption that
the person being investigated had administrative access — which is why it is
append-only, written to storage the application's own operators cannot rewrite,
often hash-chained so removing one entry breaks verification, and retained for a
period set by obligation rather than by disk cost. Open it for what an auditor
accepts as evidence and what they reject, for the difference between an artefact
that shows a control was designed and one that shows it operated across a period,
and for how to produce a complete population rather than a convincing sample.
Access Control & Separation of Duties
Almost every audit finds something here, and almost none of it is about the
access model. The model is usually fine. The lifecycle is where it breaks:
accounts that outlive employment, permissions accumulated across three internal
moves and never removed, standing production access held by people who need it
twice a year, service accounts nobody owns. This subsection covers least
privilege in practice, joiner-mover-leaver automation driven from the HR system
as the authoritative source, periodic recertification by named owners rather than
by IT, just-in-time elevation tied to a ticket, and privileged access management.
Separation of duties gets its own treatment because it is a design constraint
rather than a policy: no single person should both perform and approve a
sensitive action, whoever administers the audit store should not administer the
system it records, and the combinations that are individually reasonable but
jointly dangerous are called toxic combinations for a reason. Open it for how to
express those combinations so they can be tested continuously.
Standards & Certification
This is where most engineers first meet the subject, usually because a deal is
blocked on it, and it is the area most confidently misdescribed. This subsection
covers SOC 2, ISO/IEC 27001 and PCI DSS in the terms an engineer needs: what each
scheme actually examines, who issues the output, what the scope boundary means,
and what each one does not say. The single most useful thing in it is negative
knowledge — that a report covers a defined scope over a defined period, that
"certified" is the wrong word for some of these schemes, and that a passing
result is a statement about consistency with commitments rather than a statement
that a system is secure. Open it before you answer your first customer security
questionnaire.
Where it sits in a real business
An obligation travels a long way before it reaches your code. A regulator writes
a rule about an outcome; legal turns it into a policy the organisation commits
to; risk and compliance turn that policy into controls, meaning testable
assertions about how the organisation behaves; engineering implements each
control and, separately, produces the evidence that the implementation ran.
An auditor examines the evidence and never the intention.
Loading diagram…
The step people skip is the one between the mechanism and the evidence. A
mechanism that works but leaves no exportable record produces the same audit
outcome as no mechanism at all, which is why the arrow from D to E is drawn
separately rather than folded into implementation.
The distinction engineers miss is that a control, its implementation and its
evidence are three separate artefacts, and having two of them is worth nothing.
Layer
What it is
Example
Control
A commitment stated as testable behaviour
Production changes are approved by someone other than the author
Implementation
The mechanism enforcing it
Branch protection requiring a non-author approval; deploys only from that branch
Evidence
An artefact showing it operated over a period
An export of every merge in the period with approver, timestamp and ticket
Evidence has to cover a population, not an anecdote: the auditor asks for
every production change in the period, then samples it. If you cannot enumerate
the population you have no control, however good your review culture is. "We
always review pull requests" is a habit; "reviews are enforced by branch
protection and exported monthly" is a control.
That reshapes logging too. An audit trail is not a debug log, which may be
sampled or dropped under load; an evidentiary record must say who did what to
which record, when, and what changed, assuming the person being investigated held
administrative access — hence append-only writes to storage the application's own
operators cannot rewrite, such as object storage under an object-lock policy, and
hash-chained records so that removing one breaks verification. Access control
shifts the same way, from model to lifecycle: the recurring findings are leavers'
accounts, permissions accumulated through internal moves, and standing production
access, answered by joiner-mover-leaver automation from the HR system,
recertification by named owners, and just-in-time elevation against a ticket.
Separation of duties then keeps any one person from both performing and approving
a sensitive action, so the author is not the sole approver and whoever
administers the audit store does not administer the system it records.
Commercially, the reason all of this is funded is simpler than the machinery
suggests. Compliance sits on the revenue path. An enterprise buyer's procurement
process will not let a contract through without a security review, a subprocessor
list and usually a current report; a regulated buyer adds an audit-rights clause
and a right to be notified of incidents. So the artefacts a compliance function
produces are, in commercial terms, inputs to sales, and the deadline that
actually moves an engineering team is a renewal rather than a regulator.
The vocabulary of the first fortnight
The terms below are used constantly and precisely by the people who own this
work, and loosely by everyone else. Using them precisely yourself is one of the
cheapest credibility signals available in an interview or a first week.
Term
What it means precisely
Control
A commitment expressed as behaviour that can be tested
Control owner
The named person accountable for it operating, not the team that built it
Population
The complete set of events a control applies to in a period
Sample
The subset an auditor tests, drawn from the population
Exception
An instance where the control did not operate, recorded rather than hidden
Compensating control
A different mechanism accepted because the intended one is not feasible
Scope
The systems, products and period an examination covers
Attestation
An opinion by an examiner about assertions the organisation made
Certification
A formal grant against a standard by an accredited body
Data controller
The party that decides why and how personal data is processed
Data processor
The party that processes it on the controller's instructions
Subprocessor
A processor engaged by your processor, which the customer must be told about
Lawful basis
The stated legal ground for a specific processing purpose
Retention schedule
The rule that says how long each category is kept and why
Litigation hold
An instruction that freezes deletion for data relevant to a dispute
Three of these repay attention immediately. Controller and processor decide
who owes which duty, and a vendor that quietly behaves as a controller over
customer data has created a problem no contract clause will paper over.
Compensating control is the phrase that lets an honest engineering
constraint be accepted rather than concealed, and it works only if you propose
it before the audit rather than during it. And exception is not a failure
state: an audit report with no exceptions and a wide scope is rarer than
candidates imagine, and recording one properly is the behaviour being tested.
Who does this work
Security and compliance engineers, sometimes titled GRC engineers, build the
mechanisms and the evidence pipelines — control mapping, automated configuration
checks, evidence exports, the auditor's request list — and spend as much time
arguing whether an existing mechanism satisfies a control as writing code. It is
a role with an unusual centre of gravity: the deliverable is frequently an
argument, supported by an export, rather than a feature.
Privacy engineers own data inventories, retention and deletion machinery and
subject-request handling, alongside a data protection officer, a role the GDPR
requires of some organisations, who interprets rather than implements. GRC
analysts run the control framework itself, map one scheme's requirements onto
another's so that a single piece of evidence serves several, and chase owners.
Internal auditors test independently and report to the board, not to engineering,
which is the point of them; treating internal audit as an adversary is a
career-limiting instinct, because they are usually the cheapest way to find out
what the external examination will say. Architects in regulated industries carry
the subject as background, because their real contribution is noticing at design
time that a requirement has created a cross-border transfer.
Role
What they own
What they do not do
Security & compliance engineer
Mechanisms, evidence pipelines, control automation
Decide what the law requires
Privacy engineer
Data inventory, retention, deletion, subject requests
Give the legal interpretation
GRC analyst
The control framework, mapping between schemes, owner chasing
Build the enforcement
Data protection officer
Interpretation, regulator contact, advice and oversight
Implement, or report to the team they oversee
Internal auditor
Independent testing, reporting to the board
Design the controls they test
External auditor
The opinion, against a defined scope and period
Consult on how to fix what they found
The split to internalise is between people who specify controls and people who
implement them; when nobody does the second, the year ends in a scramble for
screenshots.
Demand, adoption and how that is changing
Demand is high for commercial rather than idealistic reasons: compliance gates
revenue. Enterprise and public-sector buyers will not sign without a security
review, so a vendor without a current report loses deals it never competes for.
The number of jurisdictions with comprehensive privacy law keeps growing, and AI
systems have added obligations around transparency, data provenance and human
oversight that reach engineers as lineage and evaluation work.
It is also where most engineers meet the two regimes, and both are widely
misdescribed. A SOC 2 report is an attestation by an independent audit firm under
AICPA standards against the Trust Services Criteria; security always applies,
while availability, confidentiality, processing integrity and privacy are in
scope only if the company chooses. Type 1 covers whether controls were suitably
designed at a point in time, Type 2 whether they also operated effectively across
a period, which is why buyers want Type 2. Nobody is "SOC 2 certified": the
output is a report holding the company's own system description, the auditor's
opinion, the tests performed and any exceptions, and it is normal both to have
exceptions and to leave products out of scope.
ISO/IEC 27001 is genuinely certifiable, by an accredited body, but it certifies
an information security management system — the process by which you assess risk,
select controls and improve — rather than any particular technical control, and
you are audited against your own risk assessment and statement of applicability.
Neither says a system is secure. Both answer a narrower question: does this
organisation do what it says, within a scope it defined, and can it prove it.
PCI DSS is different in kind again and worth separating in your head. It is a
contractual standard maintained by the PCI Security Standards Council and
enforced through the card brands and acquirers rather than by a regulator, it
applies to entities that store, process or transmit cardholder data, and the
single most consequential engineering fact about it is that scope is reducible.
Segmenting the network and keeping card data out of your systems — by tokenising
or by letting a payment provider capture the details directly — shrinks what is
in scope, which is why so much payment integration is designed to keep the
primary account number away from your servers entirely.
Evidence collection itself has commoditised, since compliance platforms and cloud
configuration rules now check control state continuously. Demand moved upstream
rather than away: the scarce skill is designing systems whose compliant behaviour
is a property of the architecture. The platforms will tell you that a bucket is
public. No platform will tell you that the feature you shipped last sprint
created a transfer of special-category data to a subprocessor in a jurisdiction
with no adequacy decision.
What makes it hard
Obligations are written about outcomes, in deliberately technology-neutral
language so they outlive implementations, so turning "appropriate technical and
organisational measures" into a mechanism is a judgement call. You improve at it
by reading how auditors and regulators treated similar arguments, not by reading
more statute. There is no lookup table from rule to mechanism, and anyone selling
you one is selling a checklist that will be wrong for your architecture.
And nearly everything here is cheap at design time and close to impossible later.
Residency, once required, turns a global database into region-pinned deployments
and forces routing decisions before the write rather than after, then breaks on
the copies you forget are copies: the search index, the warehouse, the logging
pipeline, a support tool with production access, any subprocessor. Cross-border
transfer is the related question of whether data may leave at all; under the GDPR
regimes that needs a lawful mechanism such as an adequacy decision or standard
contractual clauses, and the transfer nobody noticed is usually created by
telemetry rather than by the product.
Erasure is the sharpest case. Learn privacy regimes at the level of the duties
they create — name a lawful basis for each purpose, use the data only for that
purpose, collect no more than you need, tell a person what you hold and correct
it, delete it in defined circumstances, stop keeping it indefinitely — and the
collision is immediate. Deleting the row does not delete the data.
Loading diagram…
The interesting participant is the last one. Backups are not deleted from; they
are handled by a bounded retention window plus a documented rule that restored
data is re-processed against the deletion list, and being able to say that
sentence calmly is what distinguishes a credible answer from a fantasy about
instant physical erasure everywhere.
The workable answers are architectural. Crypto-shredding puts each subject's
data under a per-subject key you can destroy, so ciphertext sitting on immutable
media becomes unreadable without being rewritten. Pseudonymisation keeps most
systems holding tokens, so only the identity vault has to honour erasure.
Tombstoning and compaction handle keyed event logs. And automatic expiry
beats deletion on request for the simple reason that it runs without being asked,
which is also why a retention schedule is the highest-leverage privacy artefact
most organisations do not have. Some copies you are required to keep: an audit
trail you can delete from is no longer evidence.
Obligations also conflict in ways that have no technical resolution. A deletion
request meets a retention duty; a litigation hold freezes data a policy is trying
to expire; a fraud model wants history that data minimisation says you should not
keep; an anti-money-laundering duty to retain records sits directly against a
customer's request to be forgotten. You resolve those by naming the competing
bases, deciding deliberately and writing it down. The written record is the
deliverable — an unrecorded good decision and a bad decision look identical
eighteen months later.
And evidence is continuous. A control that worked in March and broke silently in
July fails a period-based examination even if it is healthy on audit day, so
control failure needs alerting exactly like service failure does. The mental
model to adopt is that a control has uptime.
Loading diagram…
Note that the path to an exception does not require anybody to do anything wrong;
it only requires drift to outlive the period. That is the whole argument for
continuous control monitoring in one transition.
The last difficulty is scope, and it is the one that produces genuine
disagreement rather than genuine difficulty. A narrower scope is easier to pass
and less useful to the buyer; a broader one is honest and expensive. Engineers
tend to assume scope is a technical fact and are surprised to discover it is a
negotiated position, argued between sales, compliance and the audit firm, with
real consequences for what you will be asked to build next quarter.
Why study it
Study it if you want to be trusted with architecture in a regulated industry, or
you work somewhere that sells to enterprises. The leverage is that very few
engineers can look at a design and say "this creates a cross-border transfer and
we have no mechanism for it" or "we cannot honour erasure here, because state is
rebuilt from an immutable log". Said at the right moment, that saves a rebuild,
and unlike a framework it is still true in ten years and in another industry.
There is a career-shaped reason too. Compliance knowledge is one of the few
things that makes an engineer legible to executives, because it connects a design
decision to a contract, a deal or a fine in one step. That is a useful property
if you want to move towards architecture or towards leadership, and it is close
to free if you are already in a regulated business, since the obligations are
being explained to you anyway.
Skip it if you want depth in algorithms, machine learning research or systems
performance, where the overlap is small. Much of the work is written reasoning
and negotiated definitions, so if you dislike producing documents other people
argue with, you will enjoy the design slice and resent the rest. And be honest
with yourself about tolerance for work whose success condition is that nothing
happens.
Your first hour
Take one piece of personal data in a system you know — an email address, a date
of birth — and trace it. Design nothing. Write down every place it comes to rest
and what would happen if its owner demanded erasure today.
Copy
How it got there
Deleted by
Retention
Gap
users row
Signup
Admin DELETE
Until deleted
—
Nightly backups
Backup job
Nothing
35 days
Restore reintroduces it
signup_completed events
Event stream
Nothing
Indefinite
No per-subject key
Warehouse dim_user
Nightly ETL
Nothing
Indefinite
Copy nobody owns
App logs
Payload logged
Log expiry
30 days
Never should have been logged
Finding four copies you had forgotten is the normal outcome, and the finished
table is what a privacy team calls a data inventory. Two columns are doing the
real work: "deleted by", which is empty far more often than anyone expects, and
"how it got there", which usually names a pipeline built for an unrelated reason
by someone who is no longer on the team.
Then take one control — "production changes are approved by someone other than
the author" — find the mechanism enforcing it, and try to export last month's
complete list of production changes with approvers. Write the export as something
repeatable rather than as a one-off query, because repeatability is the whole
difference between evidence and an anecdote.
Control CHG-01 Production changes are approved by a non-author
Period 2026-06-01 to 2026-06-30
Population every merge to the deployable branch in the period
Export id, author, approver, merged_at, ticket, deployed_at
Checks approver is never equal to author
every deploy maps to exactly one merge in the population
count of merges matches count of deploy records
Exceptions 2 emergency deploys on 14 June, approved retrospectively
Failing to produce that list in an hour is how you discover why this discipline
exists — and if you do produce it, the two emergency deploys at the bottom are
more instructive than the thirty clean ones, because how you handle a documented,
retrospectively approved exception is exactly what an auditor is trying to learn
about you.
What this is not
It is not paperwork added at the end, though it is often practised that way,
which is where the reputation comes from. Every organisation that treats it that
way pays for it in the same currency: a fortnight of engineering time each year
spent taking screenshots, and an architecture that has to be unpicked the first
time a large customer asks a residency question.
It is not security: security reduces risk, compliance demonstrates you did what
you committed to, and either can exist without the other. A system with strong
encryption, no logging and no access reviews is secure and unauditable. A system
with weak controls, immaculate records and an honest scope statement can pass.
It is not legal advice, and offering it as such is a real risk rather than a
stylistic one. The correct engineering posture is to describe the mechanism and
its properties precisely and let the lawyer or the data protection officer decide
whether those properties discharge the obligation. "Here is what the system does
and here is what we can prove about it" is always in your remit; "that satisfies
the regulation" usually is not.
And it is not a certificate you collect. A report covers a defined scope over a
defined period, which is why answering a security questionnaire by attaching a
PDF proves far less than people assume — and why reading someone else's report
properly means turning straight to the scope statement and the exceptions rather
than the opinion.
A control you cannot evidence for the whole population is not a control, it is
a habit, and habits do not survive an audit.