A maintenance script updates your permissions table and every user in one tenant can suddenly edit everything. What should have stopped that, and how would you have found out?

The escalation came through the data, not the door

Almost everything written about authorisation concerns the request path: is the token valid, does this subject hold that role, is the check in the right layer. This failure bypasses all of it. Every check ran correctly. The enforcement point behaved exactly as designed. What changed was the data the check consults, and because that data is stored in ordinary tables reached by ordinary credentials, it changed without any of the ceremony a code change would have attracted.

So the first thing to say out loud is that the permissions table is not application data. It is the executable form of your security policy, and a write to it is a privilege change. If a migration, a support tool, a backfill job or an ad-hoc session can update it with the same access used to fix a typo in a customer's address, you have a policy engine that anybody with a database connection can rewrite.

Why a stamped grant fails worse than a derived one

Trace what a bad UPDATE costs you under each of two models. In the first, entitlements are stamped: each user row or membership row carries the permissions that user holds, written at the time of some earlier decision. A script that sets a column across a tenant destroys the only copy of the truth. You now cannot distinguish the rows it touched from the rows that were legitimately in that state, because both look identical afterwards. Recovery means a point-in-time restore of that table and a reconciliation of every legitimate change made since.

In the second, entitlements are derived. The grants that matter live in a smaller, deliberately boring source of truth — a role definition, a group membership, a subscription tier — and the effective permission for a request is computed from it. A bad write to a derived cache or projection is annoying but recoverable, because you can rebuild it. The property you want is that the authoritative record of who may do what is small, append-mostly, and reconstructible, and that anything large and mutable downstream of it is a projection you can throw away.

That is also why the answer to "how do you model roles" and the answer to this question are not the same answer. A well-normalised relationship model still fails here if any process can write arbitrary rows into it. Modelling gives you expressiveness; change control gives you integrity.

Treat a bulk grant change like a deploy

The controls worth naming are the ones that already exist for code, applied to policy data. A change to entitlements should arrive as a reviewed artefact rather than as a statement someone types, which in practice means a migration or a job in version control with a second pair of eyes, running under an identity that exists only for that purpose. The write path should be an interface that understands what a grant is, so it can refuse a change that violates an invariant, rather than raw SQL that cannot.

Blast radius is the next control, and it is the one most often missing. A job that alters entitlements should be scoped to one tenant, or to a bounded set of subjects, and should be required to declare how many rows it expects to touch and abort when the count disagrees. A script that intended to fix eleven users and matched eleven thousand had all the information it needed to stop.

Then reversibility. Write the previous state before you overwrite it, so the inverse operation is available without a restore. Model an entitlement change as an event with a validity window rather than as a destructive update where you can afford it, because then revocation is a new fact and history stays intact.

Finally, no standing broad grant. A support engineer who needs to see a customer's data should get a time-boxed, logged, tenant-scoped elevation rather than a role that quietly makes their account the most dangerous object in the system.

The detection question is the one candidates skip

Assume all of the above and assume it fails anyway, because it does. The gap between the bad write and the discovery is what determines whether this is an incident or a breach, and closing it needs an assertion that runs on its own.

Useful invariants are cheap and specific. No subject holds an entitlement outside the tenant it belongs to. The count of subjects holding administrative rights per tenant is small and changes rarely, so a step change in it is an alert. The set of effective grants recomputed from the source of truth matches the projection the request path actually reads, and a divergence pages someone. Each of those is a query you can run on a schedule, and each of them would have caught this failure within minutes without anyone knowing in advance what the script was going to get wrong.

Alongside that, log the change itself. An audit record for a grant needs the subject, the entitlement, the resource, the before and after state, the actor and, critically, the actor for automated writes too. Trails that capture human console actions but not the batch job that runs at 02:00 are the trails that go blank at exactly the moment you need them.

flowchart TD
  A[Source of truth grants] --> B[Effective permission projection]
  B --> C[Request path check]
  A --> D[Recompute expected set]
  B --> D
  D --> E{Sets agree}
  E -- no --> F[Alert and freeze grant writes]
  E -- yes --> G[No action]

The edge worth noticing is that the comparison reads the same projection the request path reads, not a second derivation of it. An invariant computed from the source of truth alone tells you the policy is sound while the thing serving traffic disagrees with it.

How long the mistake outlives the fix

Correcting the row does not end the incident, and this is where the design decisions made for latency come due. If permission decisions are cached at the edge, in a sidecar, or in each service's process, the window in which a revoked grant is still honoured equals that cache's lifetime. If entitlements were embedded in an access token, the window is the token's remaining validity, and no database change shortens it. If your answer is that both are short, be ready to say what short means and how you would confirm it during an incident rather than hoping.

The escalation also has consequences you cannot roll back at all. Data read under the wrong grant has been read, and writes made under it are now indistinguishable from legitimate ones unless the audit trail records which identity and which entitlement authorised each change. That is the argument for keeping the entitlement used on the record of the write itself.

Reading the Salesforce permissions incident

In May 2019, a database script at Salesforce granted broad write permissions to users who should not have held them, and the service had to be shut down to contain it. It is a useful reminder that authorisation failures rarely arrive through the login path: the permission model itself was sound, and a maintenance job wrote to the data that expressed it. The blunt containment measure is the detail worth dwelling on, because when the only way to stop over-permissioned users from writing is to stop everyone from writing, you have discovered that you had no narrower control available. So an answer here should cover who and what is able to change a permission grant, whether such a change is reviewed, logged and reversible, and what you can switch off that is smaller than the whole service.