What an Accuracy Number Means

Every vendor quotes an accuracy figure. It is usually the least informative number available and it is frequently not comparable to anything.

Why one number fails

It depends on the base rate. A system detecting an event that occurs in one frame in ten thousand can be right 99.99 percent of the time by never detecting anything.

It hides the trade-off. Every detector has a threshold; moving it trades false alarms against missed events. A single figure has picked one point and not told you which.

It is measured on a test set, whose relationship to your site is unknown.

It averages over conditions. Good in daylight, poor at dusk, and the average is meaningless for a system that runs at dusk.

The four figures that matter

True positive rate, or recall: of the events that occurred, what proportion were detected? This is the safety-relevant number.

Precision: of the alerts raised, what proportion were real? This is the alert-fatigue number.

False alarms per camera per day: the operationally meaningful form of precision, because operators experience alerts per shift rather than a ratio.

Missed events per period: the operationally meaningful form of recall.

Ask for all four, at a stated threshold, and treat a refusal as an answer.

The trade-off is a curve, not a point

Lower the threshold: catch more, alarm more.

Raise it: fewer alarms, more misses.

The whole curve is the system's characteristic; the point you choose is an operational decision that depends on the cost of each error type.

For a safety alarm, missing an event is worse than a false alarm — up to the point where false alarms are so frequent that operators ignore all of them, at which point you have neither.

Choose the operating point deliberately, measure the alert rate that results, and revisit it. This has its own note.

Distribution matters more than the average

The point this whole field has most often got wrong.

Error rates are not evenly distributed across people. Systems involving human appearance have repeatedly been shown to perform differently across skin tone, sex, age and other characteristics.

An overall accuracy figure conceals this entirely.

A system that works well on average and poorly for a subgroup is not an accurate system. It is a system that fails specific people, and if it gates access or triggers intervention, it fails them repeatedly.

Demand subgroup performance figures, and where the vendor has not measured them, treat that as the finding. This has its own note too.

Testing on your own site

Benchmark performance does not transfer. Your cameras, angles, lighting, weather and population differ from the test set.

Run a pilot with ground truth: a period where a human independently records what actually happened, compared against what the system reported.

Expect a substantial drop from the quoted figures. That drop is the real number.

Repeat seasonally, because winter light and summer light are different problems.

The base rate calculation

The arithmetic that most often should end a procurement and rarely gets done.

Take the true event rate: how often the condition actually occurs, per camera per day.

Take the measured precision at the chosen threshold.

Compute false alarms per true detection.

For a rare event, even a high-precision system produces mostly false alarms, because precision is dominated by the number of opportunities to be wrong.

If operators will see many false alarms per real one, the system is not deployable as an alarm whatever the accuracy figure says. It may still be excellent for retrospective search.

Asking for the curve

One point on a trade-off is a choice someone else made for you.

Ask for recall and precision across a range of thresholds, not at one.

Plot it. The shape tells you how much room there is: a curve that falls steeply means small threshold changes cost a great deal, and there is no comfortable operating point.

Compare against your alert budget to find which points are usable at all.

Where a vendor supplies only one figure, ask what threshold produced it, and treat an inability to say as a finding about the product.