Let me tell you about a team I will call the Harbor group, a small crew of five who looked after the machine learning models behind a regional delivery company’s route planning. Nothing about them was unusual. They were friendly, a bit overworked, and very good at their jobs. They also had a secret problem, which they only noticed on the Monday morning when their model refresh failed and the one person who knew how to do it was in a hospital waiting room with a broken wrist.
What follows is how it played out, and how a stack of short pages, the humble runbook, changed how the team worked.
The refresh only Dana could do
Every second Monday, the team retrained the route-time model on fresh data and pushed the new version live. Dana had done it from the beginning. She never called it complicated. “It is just a few steps,” she would say, and then rattle off a sequence that nobody else could repeat, because half of it was unspoken. Check that last night’s data landed. Skip the second table if the loader complained, but only on the first Monday of the month. Never promote the model before the comparison notebook finishes, even though the button looks ready.
When Dana went offline, the team tried. Tobias ran the training script and promoted the result. The comparison step he skipped turned out to matter: the new model was slightly worse on late-evening routes, and dispatchers noticed within hours. The refresh was rolled back, an apology was sent, and a quiet, uncomfortable team meeting followed.
Dana’s idea from the hospital bed
It was Dana herself who proposed the fix, by message, from a waiting room. “I should have written it down years ago,” she said. “Let’s do it properly. Tobias, call me tomorrow and I will talk, you type.”
For two evenings, Tobias recorded the call and turned it into a first draft. What came out was three pages. Dana read the draft and found six gaps in the first ten minutes. Not because Tobias was careless, but because she had never said those parts aloud before. Each gap became a line in the document.
The shape they settled on
The team agreed on a layout and used it for every procedure they wrote afterwards:
- Why we do this and when. A sentence or two.
- Before you start. Data checks, access, and who to tell.
- The steps. Numbered, with a “you should see” note beside each.
- Stop points. The places where you pause and confirm before going on.
- Undo. How to roll back, in the same level of detail as going forward.
- Last run by. A name, a date and a one-line note on anything odd.
The first time someone else ran it
Two weeks later, Dana was back but on light duties with her arm in a cast. She decided not to touch the refresh. Mirela, the newest member of the team, who had never run it, followed the runbook line by line while Dana sat across the room and kept her mouth shut. It took a little longer than usual. Mirela paused at step seven, where the wording was ambiguous, and Dana had to bite her tongue. Afterwards they rewrote that line together.
The refresh went out without drama. More important, the team noticed something they had not expected: Mirela caught a data issue at step two that Dana would have glossed over out of habit. A fresh pair of eyes, guided by a plain checklist, had actually improved the process.
The night the data feed arrived late
The real test came in the second month. A partner’s data feed arrived four hours behind schedule on a refresh Monday, and Dana was on a train with spotty signal. Mirela and Tobias opened the late-feed page they had written only weeks before. It told them to hold the refresh, post a one-line notice in the team channel, check the feed’s arrival time every thirty minutes and, if it had not landed by noon, skip the cycle and run the refresh on Tuesday instead.
They followed it exactly. At half past ten the feed landed, they ran the refresh, and Dana, who had been bracing for a panicked phone call, got a cheerful message instead: “Done, all checks green, please do not look at your phone.” She laughed so loudly on the train that a stranger asked what was funny. That was the moment she stopped worrying about being the only one who knew.
What they added over the next quarter
Once the first runbook worked, the Harbor group got hooked. Over a few months they wrote entries for the things that had always lived in someone’s head: restoring a previous model version, rebuilding the feature table after a schema change, handling a late data feed, and pausing automated retraining during a holiday week. They also began a rule that every fix became a page. When a flaky job was finally traced to a time zone setting, the person who solved it spent ten minutes writing “symptom, cause, fix” in the shared folder before closing the ticket.
The wider discipline of running machine learning in production has a lot to say about this, as set out in a piece on how disciplined operations speed up model releases. And for teams mapping out what to learn and document first, a roadmap written for data scientists and engineers makes a useful companion. The Harbor group would say their own journey started with something simpler than any roadmap: a page that said what to do on Monday.
One small ritual helped it all stick. Every Friday, the team spent ten minutes asking a single question: “Did anything happen this week that the pages did not cover?” Most weeks the answer was no. When it was yes, somebody took the note away and had the page fixed before the weekend.
Where they went to read more
When the team wanted to see how other groups kept recurring data work dependable, one of them pointed to TheDataOps.org, a DataOps blog written for data engineers and their teams. It gave them vocabulary for what they had stumbled into, though the heart of it was still writing things down and trying them on a colleague.
What changed in the team
The obvious change was resilience. Anyone on the team could now run the refresh, and nobody planned holidays around Dana’s calendar. The less obvious changes mattered just as much.
- Onboarding got faster. New people learned by running procedures, with a safety net, instead of shadowing for weeks.
- Dana got her time back. She stopped being the only answer to every question and moved on to improving the model itself.
- Arguments got shorter. When two people disagreed about “the right way”, they edited the page and moved on.
- Mistakes got smaller. The comparison step that Tobias once skipped now had a stop point and a note explaining why it existed.
The lessons worth stealing
If you want to try this in your own team, the Harbor group’s experience boils down to a few simple moves. Pick the task that makes you nervous when the usual person is away. Have them talk through it while someone else writes. Give the draft to a newcomer and say nothing while they follow it. Fix every hesitation. Date the page, name an owner and put it where people actually look.
Dana still keeps a printed copy of the first draft pinned above her desk, the one with the six gaps scribbled in the margin. “That is the day this team grew up,” she told me, half joking. I think she is right. A team becomes sturdier not when it hires a hero but when it makes the hero’s knowledge everyone’s to use, one short page at a time.