The story below is invented for illustration. Every person, firm and figure in it is made up, though the pitfalls come straight from the practices described in the Ops Atlas entry on PatchOps, an informal label for enterprise patch management. Think of it as a parable for infrastructure groups who recognise a little of themselves in it.
The team and its unspoken habit
Picture a platform group of five engineers supporting a mid-sized analytics company. They looked after roughly six hundred servers, a mix of database hosts, application nodes and a handful of model-serving machines. Every month, when the vendors released their updates, the group did what it had always done: a senior engineer would announce “we’re patching this week”, a few people would grab batches of machines, and the work would be done by Friday. Mostly.
The word “mostly” was the problem. Nobody could say exactly how many machines were finished. The dashboard said ninety-four percent. The engineers believed it, partly because nobody had time to doubt it.
The Tuesday it went sideways
One Tuesday in the middle of the month, the customer-facing reporting service slowed to a crawl at ten past ten in the morning. Support tickets piled up within minutes. The cause turned out to be embarrassingly ordinary. A tool default had triggered a restart of several application nodes during working hours after an update had been installed the previous night. The nodes came back one by one, but the load balancer sent traffic to them before their caches warmed up.
The group spent the afternoon apologising and the evening reading. One of them found the Atlas page and passed around the paragraph about business-hours reboots from tool defaults being a known anti-pattern. It was uncomfortable to see their own incident listed in a reference article.
The retrospective that actually helped
The retrospective produced a list rather than a blame. Four items stood out.
- Nobody knew which machines were waiting for a restart.
- Restart timing came from a default, not from a decision.
- The ninety-four percent figure was built on machines that were reporting.
- There was no first group: every batch was effectively a production batch.
The group’s most junior engineer, who had been quiet through the whole meeting, drew a small diagram on the whiteboard: four boxes, each larger than the last, labelled canary, pilot, broad and careful. She had read about rings. The senior engineer asked how many machines would go in the first box. “One or two percent,” she said, “so about six.”
Building the rings, one awkward week at a time
The next month looked different. Six machines formed the canary ring, chosen so that one of each server type was present, including a model-serving node. This detail mattered more than anyone expected. A role-specific regression appears only on hardware that actually does the job, and an unpatched canary set made of idle test machines would have missed it. For a feel of why serving machines for machine-learning workloads need that kind of care, the team re-read an article on the part operations plays in machine learning projects.
The pilot ring took about forty machines, five to ten percent of the estate, and the broad ring took the rest, aside from a careful group of the database hosts, which would be handled one at a time with an engineer watching. They named a ring captain for each ring. Not a hero, just someone whose job was to say go or stop and write down why.
The capacity sum nobody had done
The second surprise was arithmetic. The group had assumed the monthly work fitted in one maintenance evening. Then they did the sum, borrowed almost line for line from the Atlas example: six hundred servers, around forty-five minutes per machine, no more than ten percent updating at once. That is two hundred and forty machines per window, so three windows to finish a cycle.
The senior engineer said nothing for a moment. Then: “We have been promising one.” They negotiated three evening windows with the business, spread over two weeks, and published the dates in the shared calendar. Complaints dropped. Predictability turned out to matter more than speed.
Gates, and the first time they said no
The group wrote simple gates for each ring: install failures below two percent, no fewer than nine in ten machines still checking in, no new incidents. In the third cycle the gate said no. The pilot ring showed a slightly elevated failure rate on one node type, a bit above the threshold. The old group would have pressed on, reasoning that a few failures were normal. The new group halted and looked. A storage driver update was colliding with a monitoring agent. They held the broad ring for two days while a corrected package arrived, then proceeded.
No customer ever knew. That is the whole point of a canary: the problems are found on six machines, and the other five hundred and ninety-four are spared.
Counting honestly, and the ugly number
The hardest conversation came in the fourth month, when the group recalculated coverage with every in-scope machine in the denominator and credited only machines that had restarted and still reported. The ninety-four percent became seventy-nine. Several machines had been silent for weeks. Two waiting-for-restart hosts had been in that state since the previous quarter. One expired exception, a legacy reporting box nobody owned, turned out to be exposed to the internet.
The temptation was to hide it. Instead the group put both numbers in the monthly report, with a short explanation of the change in method. The head of engineering read it, asked three questions and funded a small project to retire the legacy box. For teams that operate machine-learning platforms and want more reading on how operational discipline compounds, the article on bringing in outside help for machine learning operations is a decent next stop.
The emergency drill
Somewhere around month six, the group rehearsed an urgent update. They picked a quiet afternoon, pretended a serious fix had appeared, and timed how long it took to reach ninety-five percent of machines active. It took nearly two days, longer than the agreed target. The delay came from approvals, not tooling: nobody knew who could say yes. They wrote down named approvers, shortened the canary period for emergencies, and ran the drill again. This time it took half a day.
A year in, the small changes that stuck
By the following autumn the group had added a few habits that never appeared in any plan. They froze the tested package set for each cycle, so that later rings received exactly what the canaries had run, not whatever had been published in between. Deferrals went into a small ledger, every line carrying a lapse date, and the group reviewed it in the last meeting of every month. They also started recording how long each ring took, which made the next calendar easier to write.
The senior engineer admitted that the most valuable change was cultural. Before, a late machine was an embarrassment to be hidden. Afterwards, it was a row in a list with an owner, and discussing it felt as ordinary as discussing a failed test. That shift in tone cost nothing and probably prevented more trouble than any tool they bought.
What they would tell another team
- Pick a first ring of real machines, one per role.
- Do the capacity maths before promising dates.
- Choose restart windows on purpose.
- Write gates before the change, not after.
- Show the honest coverage figure next to the comfortable one.
- Rehearse the emergency path.
None of these ideas are exotic, and none are the group’s own. They come from the standards and cases collected on the Atlas page, the Ops Atlas’s PatchOps entry, worth opening if your own monthly cycle currently ends with a vague feeling that it probably went fine.
Back in our imaginary company, the Tuesday slowdown has not returned. The group still has bad days, as every group does. But the bad days now arrive on six machines, on a calendar, with a name attached to the decision. That is a modest ambition, and it is plenty.