How would you test a model for bias?

First problem: you may not have the attribute

Every fairness test compares outcomes across groups, so it needs group membership, and the awkward fact is that the attributes people most want to test across are the ones you are least likely to hold. Sex may be in the record; ethnicity frequently is not; disability almost never is. In some jurisdictions collecting a characteristic for a lending decision is restricted while measuring disparity across it is expected, putting the two requirements in direct tension.

So the first part of the answer is how you will obtain the attribute, and there are only a few honest options. Collect it voluntarily and separately from the decision path, held where the model cannot read it and used only for evaluation — the cleanest arrangement, and it needs a lawful basis and a purpose limitation to be defensible. Take it from an existing regulatory return that already requires it. Use a statistically inferred estimate, accepting that the inference has error which propagates into every disparity figure you then compute. Or engage a third party who holds the attribute and returns only aggregate comparisons.

Proxies are the fallback and they need care. Postcode, name, language preference and product channel all correlate with protected characteristics, and a disparity across a proxy is a genuine signal worth investigating. What it is not is a measurement of the characteristic, because the correlation is imperfect in both directions: a disparity across postcode may be a disparity across income, and a null result across a proxy is not evidence of no disparity across the attribute beneath it. Treat proxy findings as a prompt, state the limitation, and never report one as a fairness metric.

The related point is that removing the protected attribute from the feature set does not remove the disparity. The information survives in correlated features — a model with no sex field learns it from purchase categories and channel — so "we do not use it as a feature" answers a different question, and offering it as a fairness control is the commonest weak answer here.

Second problem: the definitions conflict, so you must choose

There are several reasonable-sounding definitions of a fair decision, and they are not simultaneously achievable. This is a mathematical result rather than a matter of taste, and knowing it is what separates a considered answer from a list of metrics.

The main definitions in play are worth stating precisely because their differences are precise.

• Demographic parity: the rate of positive decisions is equal across groups, regardless of outcome.
• Equal opportunity: the true positive rate is equal across groups, so members of each group who would have had the positive outcome are equally likely to be identified.
• Equalised odds: both the true positive rate and the false positive rate are equal across groups.
• Calibration within groups: among everyone assigned a score of 0.3, the same fraction experiences the outcome in each group, so the score means the same thing regardless of group.

The impossibility is that calibration and equalised odds cannot both hold when the base rate of the outcome genuinely differs between groups, except in the degenerate cases of a perfect classifier or equal base rates. Kleinberg, Mullainathan and Raghavan established this for calibration together with equal false positive and false negative rates; Chouldechova established the corresponding result in the risk-assessment setting. The consequence is unavoidable: satisfying one definition is a decision that disadvantages someone under another, and that decision belongs to whoever owns the outcome rather than to the person writing the evaluation. Demographic parity and equal opportunity conflict for the same reason — equalising selection rates when base rates differ requires accepting different error rates, and the reverse.

Choosing, and justifying it against the decision

Since a choice is forced, make it defensibly, by reasoning about the harm the decision causes rather than about which metric is fashionable.

If the harm is in being missed — a screening model where a false negative means an untreated condition, or a benefits model where it means someone goes unsupported — equalising false negative rates is the natural target, because the burden of the model's mistakes should not fall more heavily on one group. If the harm is in being wrongly selected, whether an investigation, an accusation or a denial, false positive rate parity is the target. If the score is consumed downstream as a probability in a pricing or expected-value calculation, calibration within groups matters most, because a score meaning different things for different groups corrupts everything built on it.

Where a legal standard exists it constrains the choice and should be named rather than reasoned around: employment and lending law in various jurisdictions has established tests for disparate impact, and compliance will have a position on which measure applies. So propose a definition with the reasoning above, document the trade-off it accepts, and have the accountable owner and the compliance function agree it in writing before launch — not after a disparity is found, when the choice of metric becomes an argument about the result.

What the report contains

The output of bias testing is a table, not a verdict, and its construction matters.

For each group in each attribute you can test, report the selection rate, the false positive and false negative rates separately, a ranking metric and a calibration comparison. Never a single accuracy figure, since accuracy hides the composition of the errors and that composition is the whole subject. Report both error types even when your chosen definition concerns only one, because the reader needs to see what the choice cost.

Attach sample size and an interval to every cell. This is where most bias reports fall over. Consider a group with two hundred examples and a five per cent positive rate: ten positives. A false negative rate estimated from ten cases has an interval spanning most of the plausible range, so it will differ from the majority group's rate in almost every evaluation, in a random direction. Report that as a disparity and the finding is pure sampling noise; report it as a bare point estimate and you have either manufactured a governance problem or concealed one, depending which way the noise fell. Set a minimum group size below which you report the interval and decline to conclude, and record "too small to evaluate" as a limitation in its own right, since it ought to prompt collecting more data.

Test intersections where sample sizes permit, since a model can be unobjectionable across each attribute separately and poor on a combination. And test the deployed decision rather than the raw score: if a threshold, an override rule or a review queue sits between score and outcome, the disparity that matters is measured at the outcome, and a fair score can still produce an unfair decision through a threshold set once for the whole population.

After a disparity is found

The instinct is to reach for a mitigation technique; the more useful first step is to establish where the disparity comes from, because the source determines what can honestly be done.

If the training labels are themselves the record of a biased historical process, the model is faithfully reproducing that process and no reweighting of the model addresses the label — that is a data problem, and it may require a different target definition entirely. If a group is under-represented, poorer performance there may be a variance problem, addressable with more data. If a feature is acting as a proxy with no legitimate relationship to the outcome, removing it is justified on the merits. If the disparity survives all of that, the remaining levers are group-specific thresholds and post-processing, which raise their own legal questions in many jurisdictions and must not be adopted quietly by an engineer.

The definitions of fairness are provably incompatible, so there is no configuration that is simply fair. The deliverable is a chosen definition, a written justification tied to the harm the decision causes, and a per-group table honest enough to show sample sizes.