Designing alarm rules that people don't mute

Every monitoring deployment has the same failure mode, and it is not the missed outage. It is the notification channel that nobody reads any more.

It happens gradually. A rule fires at 03:14, the operator checks, the stream is fine. It happens again on Thursday. Someone widens the threshold. A month later the channel has a mute on it and the team has quietly gone back to finding out about outages from the call centre. The tool is still running, still green, still billed — and completely out of the loop.

So the useful question when designing an alarm set is not "what can we detect?" It is "will anyone still believe this in six months?" We run 24 rules across 7 notification channels, and almost every design decision below came from getting one of them wrong first.

Be slow to shout, instant to go quiet

The single most important property of an alarm rule is asymmetry. Opening and closing an alarm should not use the same amount of evidence.

Ours opens on two consecutive bad observations and closes on the first good one. Once closed, a 60-second cooldown suppresses reopening.

That asymmetry is deliberate and it is worth spelling out why, because the instinct is to make it symmetric:

• Slow to open because a single bad sample is usually a sample problem, not a stream problem. One failed manifest fetch is a lost packet; two in a row is a pattern.
• Fast to close because a stale alarm is worse than a late one. An operator staring at a red row for a stream that recovered three minutes ago learns to distrust red rows.
• Cooldown after close because a stream sitting right on a threshold will otherwise open-and-close every polling interval, and a rule that fires forty times an hour is muted the same day.

If you take one thing from this piece: a symmetric alarm rule is a flapping alarm rule.

Not every rule deserves hysteresis

Having built the flapping guard, the temptation is to route everything through it. That is wrong, and the distinction is between state rules and event rules.

A state rule describes a condition that persists — the stream is unreachable, freeze is ongoing, the certificate expires in nine days. Those want strikes and cooldown.

An event rule describes something that happened once and is already over by the time you see it. Our CDN-switch detection is the clearest case: the delivery host for a stream's segments changed. That is a discrete fact. There is no "second consecutive observation" of it — the switch already occurred. So that rule runs with one strike and zero cooldown, because the thing you want is every switch on the record, not a smoothed version of them.

Applying hysteresis to an event rule doesn't reduce noise. It just loses events.

Absence of data is not good news

This is the mistake most likely to be sitting in your alarm set right now.

If a rule is written as "if the last measurement was bad, alarm; otherwise clear", then when measurements stop arriving entirely, the rule reads as healthy. The dashboard goes quiet in the most reassuring possible way, exactly when the monitoring itself has broken.

Two things fix it. First, a rule that evaluates nothing when it has no fresh observation — ours explicitly returns "no decision" rather than closing, so an absence of data can never silently clear a real outage. Second, a separate rule whose entire job is to notice the silence: if a stream hasn't been measured in 180 seconds, that is its own alarm, with its own message ("this stream hasn't been measured for 4m — its probe may have stopped").

One detail there took us a revision to get right. The dashboard already greys a stream out as not monitored after 120 seconds, and the alarm threshold is deliberately set higher than the badge. Screen before siren: a brief gap should be visible to anyone looking, and should only escalate to a notification if it persists. Alarm thresholds that match UI thresholds exactly produce a notification for every transient blip that the screen was already communicating fine.

The same rule also needs a grace period for newly added streams. Without one, adding a stream raises "never been measured" before the probe has had a chance to pick up the assignment — an alarm caused entirely by the tool's own startup sequence. Those are the most trust-destroying alarms of all, because the operator did nothing wrong and the alarm was still correct-by-its-own-logic.

"The stream is broken" and "my view of it is broken" are different alarms

If you measure a stream from more than one location, the two situations you must never merge are the stream is down and this vantage point can't see it.

Our stream-down rule only opens when no monitoring point can reach the stream. If some points reach it and others don't, that is not a stream fault — it raises a separate, lower-severity signal saying which points have gone blind and which are still fine. Collapsing those two into one alarm means a single probe's ISP hiccup pages someone about a healthy channel, which is the canonical way to get a monitoring tool muted.

The interesting part is how we got that wrong. The first version attributed the "can't see it" alarm per probe rather than per stream. It looked more precise. What actually happened in live testing was that every rule shares one strike counter per stream and alarm type, so the reachable probe's measurement kept resetting the counter that the unreachable probe's measurement was incrementing. The counter oscillated at 1 and the alarm never fired at all.

That bug is worth describing because of how it presents: not as a false alarm, but as perfect silence. Nothing in the UI indicates a rule that is quietly incapable of reaching its own threshold. If you build consensus logic across multiple observers, write a test that asserts the alarm opens under partial disagreement — the absence of noise will not tell you.

Some rules should ship switched off

A default threshold that is wrong for the customer's setup is more expensive than a rule nobody turned on.

Our segment cache-hit rule ships disabled. The reason is that the expected origin-offload ratio varies enormously with stream popularity and CDN configuration — a long-tail channel and a flagship channel have legitimately different baselines, and a stream with server-side ad insertion runs a naturally low hit rate with nothing wrong at all. Any number we pick as a default is wrong for most deployments on day one. So the rule is off, the panel charts the rate over time, and the customer sets the threshold once they can see their own baseline.

Same reasoning for TR 101 290 Transport_error counting and for the multi-CDN divergence alarm: both default to off, both are switched on deliberately by someone who knows what normal looks like for that stream. A rule enabled with a meaningful threshold beats four rules enabled with plausible ones.

The alarm you never receive

Everything above is about which alarms fire. The failure class that matters more is the alarm that fires and never arrives, and we found ours by accident.

While testing an unrelated contact form, SMTP delivery from inside the container started intermittently failing DNS resolution — getaddrinfo returning no address associated with hostname, then resolving fine a minute later. Two of four sends failed. The same code path sends alarm notifications. Which means alarm emails had been capable of vanishing silently, with the alarm correctly recorded in the database and nobody told about it.

In a monitoring product that is the worst possible bug: the product appears to work, the alarm history looks complete, and the operator's trust is being spent on delivery that isn't happening. Notification sends now retry three times with a doubling backoff, and a recovery is logged explicitly so the pattern is visible rather than invisible.

The general principle: your notification path needs monitoring as much as the streams do. Also worth having — we escalate an alarm that has been open and unacknowledged for 30 minutes by fanning it out again. An alarm delivered once, to someone asleep, is not delivered.

When the alarm engine itself degrades

The flapping logic needs shared state, which for us lives in Valkey. So: what should happen when that state store is unreachable?

We chose to bypass hysteresis and use the raw rule decision — accept the noise, keep alarming. The alternative, holding alarms until the state store recovers, means a monitoring system that goes silent precisely when its own infrastructure is unhealthy, which is exactly when you least want silence.

Degraded monitoring must be loud, and it must be visibly degraded rather than quietly wrong.

An honest limit

None of this is free, and two costs are worth stating plainly.

Hysteresis buys quiet by spending detection latency. Two strikes at a ten-second polling interval means an outage can be up to twenty seconds older than it looks. For a hard-down channel that is a real trade, and it is the reason strike count and cooldown are configurable per stream rather than global — a flagship channel and a long-tail channel do not deserve the same patience.

And none of it repairs a badly chosen metric. Hysteresis on a rule that measures the wrong thing produces a quieter wrong answer. The rules that survived in our set are the ones where an operator can read the alarm text and know what to do next; that property has protected us from more muting than any threshold tuning.

In short

An alarm set is a trust budget, and every false positive is a withdrawal. What kept ours solvent: open slowly and close immediately, treat events differently from states, never let missing data read as healthy, separate a broken stream from a broken viewpoint, ship uncertain rules disabled, and monitor the notification path itself.

If you want a second opinion on your own rule set, we're happy to walk through ours against a real feed — including the thresholds we still argue about.