A familiar complaint in machine learning teams goes like this: the model has been nearly ready for months. Each review turns up a weakness, an experiment promises an improvement, and the deployment date moves again. Meanwhile the process the model was meant to help is still running on guesswork or manual rules. What follows works through why the stall happens, what it looks like and one cure: a threshold that the team agrees on before more tuning.
The symptoms
You can usually spot the pattern without any special tooling.
- The same notebook has been the candidate release for several review cycles.
- Discussions focus on the last small gain in a metric, not on what users are waiting for.
- Nobody can say what score would be enough to go live.
- Deployment tasks, such as packaging and monitoring, keep sliding because the model is not final.
If three of the four sound familiar, the issue is probably not technical difficulty. It is the absence of a stopping rule.
Likely cause one: the metric became the goal
A metric begins as a stand-in for usefulness. Over time, the number itself becomes the thing being optimised, and gains that nobody can feel in practice get treated as progress. A model that is slightly better on a held-out set may be indistinguishable to the people who use its predictions. The fix is to ask a blunt question: what decision does this score change, and how much would it have to improve before that decision changed?
Likely cause two: the comparison point is imaginary
Teams often measure the model against an ideal rather than against what exists today. The current process may be a spreadsheet, a hand-written rule or a human judgement call. Beating that baseline by a clear margin already creates value, and each week of delay forgoes it. The honest comparison is the model versus the present alternative, not the model versus perfection.
Likely cause three: deployment is treated as the finish line
If going live feels irreversible, teams will naturally delay it. That is a design problem. A deployment that can be switched off, run alongside the old process or limited to a small slice of traffic is far less frightening. We describe the broader ideas in how a disciplined release process speeds up machine learning deployments. The more reversible your rollout, the lower the bar for readiness needs to be.
The remedy: agree the threshold first
The cure is to decide, before further tuning, what good enough looks like. A workable threshold has three parts:
- A performance floor. The score or error level below which the model would do harm or add no value.
- A margin over the baseline. How much better than the current process it must be to justify the switch.
- A watch plan. What you will monitor after launch and what result triggers a rollback or a retrain.
Write these down with the product owner. Once they are met, the model is cleared to go, and the improvement ideas move to a dated backlog. The theory behind this appears in our explanation of the role operations practice plays in making ML work, which makes the same point about shipping with safeguards rather than waiting for certainty.
Likely cause four: polish as a hedge against criticism
Another quiet driver is fear of being blamed for a visible miss. Each extra experiment feels like insurance, as though a longer paper trail will protect the team if something goes wrong. In reality the risk is simply postponed, and the cost of delay is spread thinly enough that no one notices it. Naming the cost of waiting, such as the manual work continuing in the meantime, brings the trade-off into the open.
Checking the diagnosis
To confirm that the stall is about the stopping rule and not the model, run a short test. Take the current candidate and list the three most likely failure cases in production. For each, ask whether the monitoring plan would catch it and whether the rollback works. If yes, the remaining issue is appetite for risk, not readiness, and that is a leadership conversation more than a data science one.
What to do with the leftover improvements
Ideas that did not make the cut are not wasted. Log each one with the expected benefit and the effort, then revisit them after launch with real data. Production behaviour often shows that the weakness reviewers worried about barely matters, while a different issue nobody predicted deserves attention. Deferring the polish lets reality choose the priorities.
An outside example of stopping early
Look at how directory-style sites get published. A site like BestLawyerNow presents a legal-services listing in one scrolling page, with sections for services and a blog and buttons to book a consultation. It carries no promise of completeness; it is a layout that can go live and then change. Model releases can follow the same spirit, with a known shape at launch and refinements afterwards.
Pitfalls to avoid
A good-enough threshold can be abused. Do not set it so low that it is meaningless, and do not set it after you have seen the results, which turns it into a rationalisation. Do not drop the monitoring plan to save time. And do not treat the first deployment as the last; a model that ships should be scheduled for review on a fixed date.
When incidents do occur, a calm, practised response matters more than a flawless model. Our piece on resolving IT problems faster with AIOps platforms offers more on building that response.
Summary
If a model has been nearly ready for months, diagnose before you tune. Check whether the metric replaced the goal, whether the comparison is imaginary and whether deployment feels irreversible. Then set a floor, a margin and a watch plan, and ship when they are met. The aim was never a perfect model; it was a better outcome for the people relying on the result, and that can begin as soon as the threshold is crossed.