AI Engineering · interactive

Bias and fairness

A loan model gives everyone a score and approves above a threshold. The score is honest: the same score means the same chance of repaying in both groups. Yet one threshold approves far more of one group and turns away more of the other's good customers. Move the threshold, then try to make it fair, and find that the fairness measures disagree.

thresholds: A 60 · B 60

Group Aapprove at 60

score 0 → 100 · coloured part of each bar: the share who would repay · faded: turned down

Group Bapprove at 60

score 0 → 100 · coloured part of each bar: the share who would repay · faded: turned down

measureABgapApproved55%25%30%Repayers approved70%44%26%Defaulters approved19%6%13%Approved who repay90%87%3%

bank’s result: +363 (a repaid loan earns 1, a default costs 4)

One threshold at 60: approval 55% vs 25%; a repayer's chance of approval 70% vs 44%.

Approve at a score of60

Score everyone

The model is calibrated: a score of 70 means about the same chance of repaying for anyone. Bias isn't in the score here; it's in what the groups bring to it, from history the training data recorded.

at a score of 50: 50% repay · at 70: 90% · A averages 62, B averages 50

Pick one threshold

Approve everyone at or above it. One number, applied to everyone: the fairest-looking rule there is.

approved: A 55% · B 25%

Measure each group

Three questions, three measures. Who gets approved (demographic parity)? Of people who would repay, who gets approved (equal opportunity)? Of people approved, who repays (calibration)?

approval rate: 55% vs 25% · gap 30%
repayers approved: 70% vs 44% · gap 26%
approved who repay: 90% vs 87%

Good customers in group B are turned away 281 times, against 215 in A.

Set a threshold per group

To equalize a measure, group B needs a lower threshold. Equalize approval rates and you approve more B defaulters; equalize repayers' chances and the approval rates still differ.

B’s threshold: 60 · defaulters approved: A 19% vs B 6%

Choose, and say which

When the groups' base rates differ, a calibrated model cannot have equal approval rates, equal chances for repayers and equal false-approval rates at once. That's a theorem, not a tuning problem.

Deciding which measure matters is a product and legal decision, not a modelling one. Measure all of them per group, publish which one you optimize, and look upstream at the data: the gap in scores came from somewhere.

The impossibility is from Chouldechova 2017 and Kleinberg, Mullainathan and Raghavan 2016. The distributions here are made up; the arithmetic is not.

In production

Tools and technologies

Fairness toolkits compute these measures per group; the papers prove you can't have them all.

Not tied to one language — use these names when you search docs or design a system.