An audit of notification quiet hours asks a deliberately narrow question: when your team’s notifications are silenced, does the right thing still get through, and does everything else wait? This checklist is meant to be worked through in a single sitting with the on-call lead and a team representative. Each section lists items to verify, evidence to collect and a pass condition, so that you finish with a short list of fixes rather than a general feeling that things are fine.
Before you start: scope and materials
Gather four things before the audit begins: the current written quiet-hours policy if one exists, a list of every channel through which a person can be interrupted, the on-call rota, and the last month’s record of pages and urgent messages. If any of these are missing, the first finding writes itself.
Decide what is in scope. For most operations teams the relevant channels are chat, email, ticket notifications, monitoring alerts, calendar reminders and personal phones. Write them in a column; the audit will walk down it.
Section 1: the policy itself
- Windows are written down. Evidence: a document or channel topic listing the hours. Pass: any member can state them without looking.
- Windows reflect real focus needs. Evidence: the team’s own account of when they do design, debugging or review. Pass: the windows cover at least some of those hours.
- The scope is clear. Evidence: the policy states which channels are muted. Pass: no channel is left ambiguous.
- A review date exists. Evidence: a calendar entry. Pass: the next review is scheduled.
Section 2: the exception path
This is the section most audits find problems in, because it is the part that is hardest to test by reading a document.
- A definition of urgent exists in writing. Pass: it names concrete conditions, for example customer-facing outage or active security incident, not words such as “important”.
- Each urgent condition maps to a delivery route. Evidence: a table connecting alert type to the way it reaches a person. Pass: no urgent condition depends on a muted channel.
- The route has been exercised. Evidence: a dated drill. Pass: a test alert reached the on-call person during a quiet window within an agreed time.
- The rota is current. Pass: the person listed is actually reachable and knows they are on call.
- There is a fallback. Pass: if the first responder does not acknowledge, a second contact is notified.
Record any condition that fails. A silent path to an urgent alert is the most serious finding the audit can produce.
Section 3: the muted side
This section checks that things which should wait are really waiting, and that nothing important is quietly rotting in the pile.
- Non-urgent alerts are batched. Evidence: digest settings in the relevant tools. Pass: noisy sources send summaries rather than individual notifications.
- The backlog is read on schedule. Evidence: people confirm a routine for the end of each window. Pass: messages are not left for days.
- Senders know the rhythm. Pass: colleagues and partners can say when to expect replies.
- Misrouted urgent items are logged. Evidence: a short list of cases where something urgent was sent to chat. Pass: each case has led to a rule change or a reminder.
Section 4: a worked sample from a data and ML team
To make the items above concrete, consider a team that runs machine-learning pipelines. Their retraining jobs finish at awkward hours, evaluation reports arrive in a shared channel, and a serving endpoint powers a customer-facing feature. In an audit, the pass conditions look like this: the endpoint’s error alert is routed to the on-call phone; the retraining job’s completion notice goes to a digest; evaluation reports land in a shared room that people open when each window closes. A failure might be that a job crash is only posted to the channel, which is muted overnight.
Teams looking at that world will find useful background in the article on the part MLOps practices play in successful ML work, which helps when deciding which pipeline events are really customer-impacting.
Section 5: incident tooling interaction
- Automated correlation is accounted for. If a platform groups alerts or suggests causes, check that its notifications obey the same windows and routes.
- Escalation timers still fire. Pass: unacknowledged pages escalate even in a quiet window.
- Post-incident notes capture notification gaps. Pass: every review asks whether the right person was reached quickly.
For teams adopting such tooling, this article on speeding up resolution of IT problems with modern operations platforms provides a view of how alerts get grouped and routed, a subject that overlaps directly with the checks in this section.
Section 6: people and fairness
- On-call load is distributed. Evidence: the last quarter’s rota. Pass: no individual carries a disproportionate share.
- Personal devices are respected. Pass: the team does not expect replies on personal phones outside agreed channels.
- New joiners are briefed. Pass: onboarding covers the windows and the exception path.
- Feedback is collected. Evidence: a short, regular survey or retrospective item. Pass: concerns are recorded and acted on.
Section 7: learning time
An often-overlooked purpose of quiet windows is development. Engineers need uninterrupted time to study, whether the subject is a new language, an architecture pattern or a neighbouring discipline. If the team is curious about structured training in other technical domains, Wingversity, a website laying out an e-learning platform around aviation careers, is one example to browse; the site itself notes it is a preview carrying sample content. Check that your policy leaves room for such study, and that managers treat it as legitimate use of protected time.
Section 8: evidence log
The value of an audit depends on the trail it leaves behind. Keep a simple log, one row per checked item, with four fields: what was checked, how it was checked, what was observed and who confirmed it. Screenshots of tool settings and the timestamp of a drill alert are enough. The log lets a future reviewer see what was actually tested, as opposed to what was assumed, and it protects the team from the common situation where a setting is changed months later without anyone noticing.
Findings that turn up again and again
- The muted channel that carries something urgent. A monitoring integration posts to chat because it was configured years ago, and nobody revisited it when quiet hours began.
- The rota that lists someone on leave. The schedule is correct on paper but nobody updated it after a change of plans.
- The exception that grew. What began as three conditions has become a dozen, and the quiet windows are now quiet in name only.
- The silent sender. A partner team does not know the windows exist and keeps chasing people by direct message.
- The untested phone path. The phone integration works for the person who set it up and nobody else.
Treat each of these as a prompt for a specific check during the session, rather than waiting for them to appear by themselves.
Scoring the audit
Keep scoring simple. Mark each item as pass, partial or fail, and give each failed item an owner and a date. Do not average the marks into a single score: a team can pass nearly every item and still fail on the one that matters, an untested route for urgent alerts. Instead, sort findings into three buckets.
- Fix now: anything that could cause a missed urgent alert.
- Fix this quarter: process gaps such as a stale rota or missing onboarding.
- Watch: items that pass today but depend on one person’s habits.
Choosing participants
Keep attendance modest: the on-call lead, one engineer who is not on call, and someone from a team that depends on yours, such as support or product. The outside voice matters because it reveals whether the people who send you messages understand the windows. Allow around ninety minutes, share the checklist in advance, and ask each participant to bring one example of an interruption that felt wrong, either too loud or too quiet. Those examples often reveal more than the checklist does.
Follow-up actions after the audit
- Share the findings with the whole team, including what passed.
- Schedule the drill for any untested route within two weeks.
- Update the written policy with every change decided in the session.
- Put the next audit in the calendar, quarterly for a new policy, twice a year once it has stabilised.
The first audit usually takes longer than expected, since it uncovers routes nobody remembered. Later audits are quick, because the evidence is already collected. Keep a copy of each one so you can see whether the same findings recur, which usually points to a structural cause rather than forgetfulness.