Design an experimentation and feature-flag platform: assignment, exposure logging and a kill switch.
Evaluate feature flags in an SDK from a cached ruleset, assign users by hashing a per-experiment salt with the user id so assignment is sticky, log exposure where the arms first diverge, gate readouts on an automated SRM check, and make the kill switch a top-precedence rule that works with the control plane down.
6 min readUpdated 2026-10-10
0
Answer
Short answer
An experimentation platform has three hard parts: deciding who sees what, recording who actually saw it, and turning something off fast. Feature flags are evaluated in process by an SDK holding a cached ruleset, so a check costs microseconds rather than a network round trip. Assignment comes from hashing a per-experiment salt with the user id, which makes it deterministic and sticky. Exposure is logged where the arms first diverge, every readout is gated on an automated sample ratio check, and the kill switch is a top-precedence rule backed by a safe coded default.
Was this answer helpful?
Likely follow-ups
→A product manager wants to change a running test from 50/50 to 70/30. What happens to the users already assigned, and what do you tell them?
→A user is assigned while logged out and then signs in on another device. How do you keep their variant stable?
→How would you run two experiments on the same checkout page without one contaminating the other?
→The kill switch was pressed but one service kept serving the variant for ten minutes. Where do you look?
Pin down the load before choosing anything, because an experimentation platform sits on the request path of every service that uses it. Assume, for the sake of the design, 50 million daily active users, 30 experiments live at once, each eligible for about a third of users, and a typical request that evaluates 20 flags.
Exposure volume follows from those assumptions. If each user is logged once per experiment per day, that is 50M x 30 x 1/3 = 500 million exposure events a day, or about 5,800 a second on average. Allow three times that at peak, around 17,000 a second. At roughly 100 bytes per event, that is about 50 GB a day before compression. Now drop the deduplication: if each user hits each flag 40 times a day, you get 20 billion events, forty times more, for no extra information. So deduplication in the SDK is a requirement, not an optimisation.
Evaluation latency follows too. If each flag check were a 2 ms remote call and a request made 20 of them in sequence, flags alone would add 40 ms. That rules out remote evaluation on the request path.
Evaluating feature flags in the SDK
The control plane stores feature flags and experiments as one versioned ruleset. Each service runs an SDK that downloads the ruleset, keeps it in memory, and receives updates over a streaming connection, with polling as a fallback. An evaluation is then a lookup and a hash in local memory.
For browsers and mobile apps, avoid shipping the full ruleset. Targeting rules can reveal unreleased features and user segments. Evaluate on the server for that user and send down only the results, ideally in the first page payload so the interface does not flicker from one variant to another.
Loading diagram…
The edge to watch is between the control plane and the SDK: it is the only network dependency, and nothing to its right needs it in order to keep serving.
Assignment by salted hash
Assignment must be deterministic, so the same user gets the same variant on every request and every server without a lookup. Hash the experiment's salt together with the unit id, map the result to one of 10,000 buckets, and assign by bucket range.
import hashlib
defbucket(salt: str, unit_id: str, buckets: int = 10_000) -> int:
# A stable cryptographic digest, not Python's built-in hash(),# which is randomised per process for strings.
digest = hashlib.sha256(f"{salt}:{unit_id}".encode()).hexdigest()
returnint(digest[:8], 16) % buckets
# In the experiment if bucket < 1,000 (10% of traffic). The variant# comes from a second hash with a different salt, so widening# traffic to 20% keeps every existing user in their original arm.
in_test = bucket("exp-142-traffic", user_id) < 1_000
variant = "B"if bucket("exp-142-variant", user_id) < 5_000else"A"
A separate salt per experiment matters. If every experiment used the same hash of the user id, the same users would land in treatment across all of them, and the experiments' effects would be confounded. Distinct salts make overlap between experiments random.
Widening traffic is safe with this design, because the users already in the test stay in. Changing the split between arms is not. Moving from 50/50 to 70/30 reassigns some users from A to B mid-test, and their data now mixes both experiences. The platform should refuse that edit and offer to start a new experiment with a new salt.
Two cases need more than a hash. When a user moves from a logged-out cookie id to a signed-in account id, the hash input changes, so either randomise on the account only after sign-in or persist the earlier assignment and map it across. And when two experiments must never overlap, such as two changes to the same button, put them in one layer that shares a hash and splits its bucket range between them.
Exposure logging
An exposure event records unit id, experiment, variant, timestamp and the ruleset version that produced the decision. The version lets the pipeline detect a period when different servers held different rules. The analysis joins metric events to each user's first exposure and counts only behaviour after it.
The SDK deduplicates exposures per user, experiment and variant within a window, and the pipeline deduplicates again by key, because retries and multiple SDK instances will produce repeats.
The SRM gate
Every readout runs a chi-square test of observed arm sizes against the intended split, overall and per platform, and refuses to show results when it fails. Making it a gate rather than a chart means nobody has to remember to look. What a failed check means, and how to find the cause, is on the sample ratio mismatch page.
The kill switch
A kill switch is a rule at the highest precedence in the ruleset that forces a flag to its safe value for everyone. It must reach every SDK within seconds, so it rides the same streaming push as any rule change, with polling as the slower backstop. Pressing it is audited and restricted to named roles. The readout should also record the kill time, so analysis stops counting exposures after it.
The harder question is what happens when the control plane itself is down. An SDK that already has rules keeps serving the last version it received. An SDK starting cold with no rules falls back to the default written in code. That default is the kill switch of last resort, so it must be the safe, existing behaviour, never the new variant. A flag whose coded default is "on" cannot be killed during an outage.
Feature flags used for releases rather than experiments share this machinery. The deployment pipeline page covers flags as a rollback path and the cost of leaving them in the code.
Where exposure is logged decides whether the numbers mean anything
The design choice that separates a strong answer is the placement of the exposure event, and it is the part of an experimentation platform most often drawn as a single box. Log it at assignment, when the SDK first computes a variant, and you count users who never reached the changed screen. Their behaviour is identical in both arms, so the effect is diluted and the test loses power.
Log it inside the variant's own code path, after the new component renders, and you create the opposite problem. If the new component loads more slowly or crashes, some treatment users never log, the arms become unequal, and the SRM gate fires. The correct placement is the point where the two experiences first differ, logged by the same code in both arms. Control users are logged at the moment they would have seen the change.
Assignment is the easy part of an experimentation platform. The design is judged on where exposure is logged, whether readouts check sample ratio automatically, and what the SDK serves when it cannot reach anything at all.