Evaluating It Properly

Most reported performance in this field comes from a validation set that was used to tune the system. That is not an evaluation.

The held-out set

Data the model never saw, and that was never used to choose a threshold, a model version or a hyperparameter.

Contaminated the moment you look at it and adjust, which happens gradually and unconsciously.

Keep two: one for iteration, one touched only at the end.

Collected at a different time from the training data, which catches temporal effects that a random split hides.

Random splits of a video dataset are misleading, because adjacent frames are nearly identical and end up on both sides of the split, producing an accuracy figure that reflects memorisation.

Ground truth

Independent of the system, recorded by a person who is not looking at the system's output.

With a defined labelling standard, the same one used for training.

Covering the difficult conditions, not a convenient afternoon.

Long enough to contain the rare event you care about, which for a rare event is a long period.

This is the expensive part of evaluation and the part that gets cut, which is why so many deployments have no real performance figure.

What to report

Recall and precision at the chosen threshold, separately.

The whole curve, not one point, so the trade-off is visible.

False alarms per camera per day and missed events per period, which are the operational forms.

By condition: hour of day, weather, camera, zone.

By subgroup where people are involved, with the evaluation set composition stated.

Sample sizes, because a subgroup with thirty examples supports no conclusion.

The base rate trap

A system detecting a rare event will produce mostly false alarms even at high precision, and this surprises people every time.

Work the arithmetic before deployment, using the true event rate and the measured error rates. The number of false alarms per true detection is the number that determines whether anyone will act on the output.

If the answer is that operators will see many false alarms for each real one, the system is not deployable as an alarm, whatever the accuracy figure says. It may still be useful for retrospective search.

Piloting

On your cameras, in your conditions, for long enough to include the hard periods.

With ground truth collected in parallel.

Without the vendor tuning during the pilot, or you are measuring their tuning rather than the product.

Reported honestly, including where the result is that it does not work well enough.

A pilot that cannot fail is a procurement exercise with extra steps.

Re-evaluating

On a schedule, because performance drifts.

After any change: model version, threshold, camera, lighting, process.

With the results recorded so the trend across evaluations exists.

Reported to whoever relies on the system, since they are currently assuming it still works at the number they were told at installation.

The contaminated split

The most common evaluation error in video work, and it inflates every reported figure.

Adjacent frames are nearly identical.

A random split puts frame 1000 in training and frame 1001 in test.

The model is then evaluated on data it effectively memorised, producing an accuracy figure that does not survive deployment.

Split by time, by day or by scene, never by frame.

Better still, collect the evaluation set on different dates entirely, which also catches the temporal effects a same-period split hides.

Budgeting for ground truth

The part of evaluation that gets cut, which is why so many deployments have no real number.

A person reviewing footage independently, recording events against the agreed definition.

Both directions: what was reported and what actually happened, including what was missed.

Long enough to contain the rare event, which for a rare event is a long period.

Costed as a line item in the project, not absorbed into someone's week.

Without it you have impressions, and impressions are what the vendor's demonstration also provides.