Skip to content

All notes  /  Operations

Alert Fatigue

Every warning system degrades with volume. The arithmetic that predicts it, and the design choices that prevent a system from becoming background noise.

Analysis

A detection system's value is entirely dependent on someone reacting to its output. That reaction has a shelf life measured in false alarms.

The arithmetic

Compute the alert volume before deployment.

False alarms per camera per day, times cameras, gives alerts per day.

Divide by shift hours to get the interval between alerts.

A rate that seems small per camera becomes unmanageable at scale, and this is the calculation that most often should have ended a procurement and did not.

Do it with the vendor's own figures. It takes a minute and it is checkable.

What happens as volume rises

Operators triage, which means categorising without examining.

Then they batch, dealing with alerts when convenient rather than promptly.

Then they dismiss without reading, which is a rational response to a channel that is mostly noise.

Then the real one arrives and is dismissed with the rest.

This is not a discipline failure. It is what happens to anyone facing a low-precision signal repeatedly, and designing as though it will not happen guarantees it.

Reducing volume

Raise the threshold, accepting the missed events, deliberately and with the trade recorded.

Restrict zones. Detection across the whole frame produces alerts from areas nobody cares about; a defined zone removes most of them.

Restrict times. A vehicle-aisle detector does not need to run when the site is closed, or does need to run only then.

Require persistence. An object present for several seconds rather than one frame removes a large class of transient false positives.

Suppress duplicates. One person walking through a zone should produce one alert, not forty.

Combine conditions. Two signals agreeing is far more precise than either alone.

Differentiating severity

Not all alerts deserve the same interruption.

Immediate: a person in a machine zone. Interrupts.

Prompt: a blocked exit. Appears in a queue.

Informational: a count anomaly. Appears in a report.

A single undifferentiated stream trains people to treat everything as informational, which is the worst outcome.

Measuring it

Alerts per operator per shift, which is the number that predicts fatigue.

Time to acknowledge, trended. Rising means the channel is being deprioritised.

Dismissal rate without action.

Time spent per alert. Seconds means nobody is looking.

Treat a high alert rate as a defect in the system, recorded and worked on, rather than as something operators should cope with.

The review that keeps it honest

Quarterly, per rule: how often did it fire, and how often was it right?

A rule firing constantly and never right should be removed, not tuned indefinitely.

A rule that has never fired is either misconfigured or protecting against something that does not occur, and should be tested deliberately.

Removing rules is the maintenance activity that keeps the remaining ones read, and it is the one nobody schedules.

The honest position

A system whose alerts are ignored is not a control.

It is providing assurance that is not there, which is worse than having no system, because decisions are being made on the assumption that something is watching.

If the alert volume cannot be brought within what operators can handle at an acceptable miss rate, say so rather than deploying and hoping.

The zone restriction

The cheapest reduction in alert volume and the most often unused.

Detection across the whole frame produces alerts from areas nobody cares about.

Define the zone that matters — the aisle, the doorway, the machine footprint — and detect only there.

Frequently removes most false alarms in one configuration change.

Restrict by time as well: a detector for an area that is empty overnight need not run then, or need run only then.

Review the zones after any change to the space, because a moved rack invalidates the boundary.

Reviewing the rules

The maintenance activity that keeps the remaining alerts read.

Per rule, quarterly: how often did it fire, and how often was it right?

A rule firing constantly and never right should be removed, not tuned indefinitely.

A rule that has never fired is misconfigured or protecting against something that does not occur; test it deliberately.

Removing rules is what keeps the survivors credible, and it is the activity nobody schedules.

Report the rule count over time. A set that only grows is one nobody reads.