{"id":11784,"date":"2026-11-08T22:30:00","date_gmt":"2026-11-08T22:30:00","guid":{"rendered":"https:\/\/www.cotocus.com\/blog\/?p=11784"},"modified":"2026-11-08T22:30:00","modified_gmt":"2026-11-08T22:30:00","slug":"model-refresh-runbook-story","status":"publish","type":"post","link":"https:\/\/www.cotocus.com\/blog\/model-refresh-runbook-story\/","title":{"rendered":"Dana’s Broken Wrist and the Runbook That Saved Refresh Monday"},"content":{"rendered":"

Let me tell you about a team I will call the Harbor group, a small crew of five who looked after the machine learning models behind a regional delivery company’s route planning. Nothing about them was unusual. They were friendly, a bit overworked, and very good at their jobs. They also had a secret problem, which they only noticed on the Monday morning when their model refresh failed and the one person who knew how to do it was in a hospital waiting room with a broken wrist.<\/p>\n

What follows is how it played out, and how a stack of short pages, the humble runbook, changed how the team worked.<\/p>\n

The refresh only Dana could do<\/h2>\n

Every second Monday, the team retrained the route-time model on fresh data and pushed the new version live. Dana had done it from the beginning. She never called it complicated. “It is just a few steps,” she would say, and then rattle off a sequence that nobody else could repeat, because half of it was unspoken. Check that last night’s data landed. Skip the second table if the loader complained, but only on the first Monday of the month. Never promote the model before the comparison notebook finishes, even though the button looks ready.<\/p>\n

When Dana went offline, the team tried. Tobias ran the training script and promoted the result. The comparison step he skipped turned out to matter: the new model was slightly worse on late-evening routes, and dispatchers noticed within hours. The refresh was rolled back, an apology was sent, and a quiet, uncomfortable team meeting followed.<\/p>\n

Dana’s idea from the hospital bed<\/h2>\n

It was Dana herself who proposed the fix, by message, from a waiting room. “I should have written it down years ago,” she said. “Let’s do it properly. Tobias, call me tomorrow and I will talk, you type.”<\/p>\n

For two evenings, Tobias recorded the call and turned it into a first draft. What came out was three pages. Dana read the draft and found six gaps in the first ten minutes. Not because Tobias was careless, but because she had never said those parts aloud before. Each gap became a line in the document.<\/p>\n

The shape they settled on<\/h3>\n

The team agreed on a layout and used it for every procedure they wrote afterwards:<\/p>\n