Where the Errors Fall

An overall accuracy figure describes the average person in the test set. Nobody is the average person, and the deviations are not random.

The established finding

Face analysis systems have been shown, repeatedly and by independent evaluation, to perform differently across demographic groups.

Error rates have been found substantially higher for darker skin tones, for women, and at the extremes of age, with the largest gaps in the intersections.

The magnitude varies by system and has narrowed in some products, which is a reason to demand current measurements rather than to assume the problem is solved.

It is not confined to faces. Pedestrian detection, pose estimation and anything trained on images of people inherit the composition of their training data.

Why it happens

Training data composition. Models learn what they were shown, and image datasets have historically over-represented some populations.

Physics and sensor design. Camera exposure and dynamic range have historically been tuned for lighter skin, which affects the input before any model sees it.

Evaluation composition. A test set with the same imbalance as the training set will not reveal the gap.

Threshold selection on an unrepresentative validation set produces an operating point that suits the majority.

Why an average conceals it

A system at 97 percent overall can be at 99 for one group and 88 for another.

The person in the second group experiences a system that fails them repeatedly, while the reported figure looks excellent.

If the system gates something — access, scrutiny, intervention — the burden falls on the same people every time.

This is a fairness problem and an accuracy problem simultaneously, and treating it as only the first lets people dismiss it as politics.

What to demand

Performance broken down by subgroup, on a test set that actually contains those subgroups in usable numbers.

Both error types separately, since a system can have equal overall accuracy with very different false positive distributions.

The composition of the evaluation set, stated.

Confidence intervals, because a subgroup with thirty examples supports no conclusion.

Where the vendor has not measured this, that is the finding, and it should be recorded as an unquantified risk rather than an unknown.

Measuring it yourself

Hard, because measuring it requires the very attributes you should not be collecting.

Options: evaluate on a public benchmark that reports subgroup performance; use an independent evaluation where one exists for the product; or measure proxy outcomes — who is flagged, who has to retry, who is subject to manual override — without attaching demographic labels.

The last is frequently the practical route: if one group of staff or customers is repeatedly failed by the system, that shows up in complaints and in override rates before it shows up in a fairness audit.

Watch the override rate. A high manual-correction rate concentrated anywhere is a signal, and it is available from operational data.

What to do about a gap

Do not deploy a gated decision on a system with an unmeasured or large gap.

Where the application is not gating — counting, aggregate flow — the consequence is a data quality issue rather than a harm, and should still be reported.

Adjust the operating point per context, not per group, which is a different and much more fraught intervention.

Add human review at the point of consequence, which is the mitigation that actually works, and which has its own note.

Watching the operational proxies

Measuring subgroup performance directly requires attributes you should not be collecting. These proxies do not.

Override rate by area and shift. A concentration is a signal.

Repeat flags on the same individuals, which is visible without recording why.

Complaints, which arrive before any audit finds anything.

Retry rates where the system gates access.

Manual correction frequency at human review.

A cluster in any of these is data, and treating it as noise is how an uneven system runs for years.

What to do with an unmeasured gap

The common case: the vendor has not measured it and you cannot easily either.

Record it as an unquantified risk, explicitly, rather than as an unknown.

Do not deploy it on a gated decision — access, refusal, scrutiny — while it is unquantified.

Add human review at the point of consequence, which is the mitigation that works regardless of the cause.

Watch the operational proxies and treat a cluster as evidence.

Revisit at every renewal, because independent evaluations appear and vendors improve, and an unquantified risk accepted once should not be accepted permanently.