Pull up a chair, because this is the question we hear most often from teams who have just lived through a rough incident: “How did we spend six hours on that?” Usually the honest answer is that nobody ever decided when to stop trying things alone. A stop rule is a limit agreed in advance, a number of attempts, a clock reading or a symptom in the monitoring, that triggers a specific next step. Stop rules for troubleshooting prompt the same handful of questions every time, so here they are, answered the way we would over coffee.
What exactly is a stop rule in troubleshooting?
It is a short agreement, written before the problem arrives, that says “when X happens, we do Y”. X might be three failed fixes in a row, a graph that keeps climbing after a restart, or the point where the on-call engineer has been working alone for a set stretch. Y is the move that follows: roll back the change, bring in the owner of the affected service, or declare the issue bigger than one person. The rule does not solve the problem. It protects the problem-solving from tunnel vision.
Why can’t a good engineer just use their judgement?
They can, and mostly they do. But judgement has a weakness: it gets worse exactly when you most need it. Late at night, with a customer waiting and a theory half-proved, the mind favors “just one more try”. A limit agreed during a calm planning session is wiser than a decision made during the incident. Think of it as a note the calm version of you leaves for the frazzled one.
How many attempts is a sensible limit?
There is no universal number, though three is a common starting point for repeatable problems such as a failing build step or a service that will not start. For harder, open-ended issues, a count makes less sense than a checkpoint: “at the end of this stretch, we review with a second person and decide whether to continue.” Start with what feels slightly uncomfortable, then adjust after a few real cases.
What happens once we hit the limit?
Three things, in this order. First, put things back into a known state if you can, using the last working configuration. Second, write down what you tried and what you saw, even if it is rough. Third, pass it on to the named owner, with the notes. Skipping the second step is the usual mistake, because without notes the next person repeats your first hour.
Does this only apply to outages?
Not at all. We use the idea in training sessions when a lab exercise stalls: after a set number of attempts, the learner checks the reference answer or asks the instructor. Teams use it for slow code reviews, for tooling experiments that drift, and for migrations that keep hitting the same wall. Anywhere effort can silently stretch, a limit helps. If you are interested in how fast-moving teams tackle faults, this article on speeding up how IT problems get resolved shows a related angle.
Isn’t it risky to stop too early?
It can feel that way, which is why a good rule includes a way back. If you hand a problem over at the limit and the new person picks the same thread up and finishes it, you have lost very little. If you keep going alone and burn the whole night, you have lost a lot. The asymmetry favors stopping. You can also tune the rule: if it fires too early too often, lengthen it a little and see.
How do we write one down so people actually follow it?
- Use a plain sentence with a number or a signal in it, not a vague intention.
- Name the next action and the person or role who receives the handoff.
- Put it where the work happens: the runbook header, the incident channel topic, the ticket template.
- Say it out loud at the start of the session, so everyone shares the same limit.
What about data and machine-learning work?
Experiments are a perfect home for limits, because there is always another parameter to tweak. A team might decide on a fixed number of training runs before reviewing results, or a signal such as a metric that has stopped improving across several attempts. When planning that kind of work, a structured outline such as this planning outline that data scientists and engineers can share is helpful for deciding where the checkpoints belong.
Should reading and research have limits too?
Definitely. Engineers fall into the same loop with documentation and reference sites. Directory-style pages, such as BrainSurgeryHospitals, a medical reference site full of hospital listings and explanatory guides, show how easily a well-organized index invites hours of browsing. Decide the question you came with, read until you can answer it, and leave. That habit transfers directly to reading vendor docs or old incident reports.
How can a team adopt this without it feeling like paperwork?
Start small and make it theirs. Pick one recurring kind of problem, ask the team to propose a limit, and try it for a month. People accept rules they helped write. Keep the wording short, review it in a retrospective, and drop any rule that never fires or always fires, since those are signs of a bad setting. Over time you will build a handful of limits that people quote without thinking.
One last question: what is the single easiest first step?
Write one stop rule for the next problem you expect to face, tell a colleague what it is, and stick it somewhere visible. That is it. You do not need a framework, a tool or a meeting. You need a number, a next step and a person who knows. The first time the rule fires and saves you an evening, you will understand why experienced teams swear by them.