Pager noise rarely announces itself as a problem. It shows up instead as a set of symptoms that leaders misread: an engineer who acknowledges a page and goes back to sleep, a severe outage that ran for forty minutes before anyone looked properly, a rota that nobody volunteers for. Each is usually blamed on people. In most engineering organisations we encounter, the real cause is structural. The on-call channel carries so many low-value interruptions that responders have rationally stopped trusting it. This article sets out a diagnosis in the order a consultant would work through it: observed symptom, underlying cause, and the remedy that addresses it.
Diagnosing pager noise: start with the evidence
Before changing any rule, collect four weeks of paging history. For each page, record when it fired, which rule triggered it, who acknowledged it, how long acknowledgement took, and whether any change was made as a result. Even a spreadsheet is sufficient. This dataset turns a vague complaint (“on-call is exhausting”) into something that can be analysed rule by rule.
Three ratios usually tell the story: pages per shift, the share of pages that led to an action, and the share that fired outside working hours. Teams are often surprised by how lopsided the results are, with a small number of rules generating the majority of interruptions.
Symptom 1: Pages are acknowledged but not investigated
Cause. Responders have learned that most pages resolve themselves. A brief spike in latency, a transient connection error, a health check that fails once and recovers: these trigger pages, then clear before anyone can act. After enough repetitions, acknowledging without looking becomes the efficient choice.
Remedy. Require a condition to persist before it pages. A rule that fires only after a problem has lasted several minutes, or has affected a meaningful share of requests, filters out the self-healing blips while still catching sustained faults. Short-lived anomalies can be recorded for later review rather than waking anyone.
Symptom 2: One incident produces a storm of pages
Cause. Alerts are defined per component with no awareness of how components relate. When a shared dependency such as a message broker or a DNS resolver degrades, every service that relies on it raises its own alarm. The responder receives a burst of pages, each pointing at a different victim, none pointing at the culprit.
Remedy. Introduce grouping so that alerts sharing a time window and a dependency chain arrive as one incident. Where the topology is known, suppress downstream alerts while the upstream one is active. This single change often removes a large fraction of total page volume during real incidents, precisely when clarity is most needed. A more structured approach to running incidents with reliability engineering practices and automation builds naturally on this foundation.
Symptom 3: Severe outages are noticed late
Cause. The serious alert was present but indistinguishable from the routine ones. When every page uses the same channel, sound and urgency label, a customer-facing outage looks identical to a disk at seventy-five percent. Responders triage by habit, and habit says “probably nothing.”
Remedy. Separate urgency tiers by channel. Reserve the interrupting page for conditions with direct user impact or imminent data risk. Send everything else to a ticket queue reviewed in working hours. When the pager only sounds for genuine trouble, response time on real outages typically improves without anyone being asked to try harder. Faster detection is central to reducing downtime and protecting the user experience.
Symptom 4: Nobody wants to join the rota
Cause. The cost of being on call is driven by interruption count, especially overnight, rather than by how many real incidents occur. A rota that delivers several night pages a week, most of them non-actionable, will be avoided regardless of compensation or goodwill.
Remedy. Retire rules with a near-zero action rate. If a rule has fired dozens of times in a month and never prompted a change, it is not providing protection. Delete it, raise its threshold, or convert it to a daily report. Review the remaining rules on a fixed cadence and assign each one an owner who is accountable for keeping it useful.
Symptom 5: Alerts describe causes nobody can act on
Cause. Rules were written around whatever metrics were easy to collect rather than around outcomes users experience. CPU percentage, memory use and queue depth are useful context but poor triggers on their own, because high values often reflect normal busy periods.
Remedy. Page on symptoms that reflect customer experience, such as failed requests, elevated error responses or stalled transactions, and keep resource metrics as supporting detail inside the incident. Each page should also link to a short runbook stating what the responder should check first.
Applying the remedies in order
The sequence matters. A practical order of work is:
- Gather the paging history and calculate the action rate per rule.
- Retire or downgrade the rules that never lead to action.
- Add persistence conditions to rules that fire on transient blips.
- Introduce grouping and upstream suppression for shared dependencies.
- Split urgency tiers into separate channels with a written definition of each.
- Rewrite remaining triggers around user-facing symptoms and attach runbooks.
Teams that follow this order usually see the largest reduction in the first two steps, which also builds goodwill for the more involved work that follows.
Where automated correlation helps
For larger estates, manual grouping rules become difficult to maintain, and some organisations add automated correlation that learns relationships between signals. Engineers evaluating that route can find discussion of correlating and filtering alerts on TheAIOps.com, an AIOps-focused operations blog. Automation of that kind works best on top of the clean-up above, not as a substitute for it; feeding a noisy, poorly owned rule set into a smarter engine tends to produce smarter-looking noise.
The outcome to aim for
A healthy on-call channel is one where every page is worth reading and responders believe that. Trust comes back when interruptions are rare, owned by a named person and paired with an obvious first move. Once it returns, the behaviours that looked like people problems, slow acknowledgement, missed outages, rota avoidance, tend to resolve on their own, because the cause was never the people.