The quarterly review had been going for twenty minutes when Ana realised nobody in the room was talking about the same model. Her data scientists said the fraud model was “in production.” Her platform engineers said it was not, because it was only scoring a shadow copy of traffic. The business owner wanted to know why “accuracy” had dropped, and three people answered with three different numbers. Nothing was broken. The team simply lacked an MLOps team glossary: a short, shared record of what its key words mean and who looks after each one. This is the step-by-step story of how they built one in two weeks, and how you can do the same.
Step 1: Catch the words that keep causing trouble
Ana did not start with a blank document. She started with her calendar. For a week she kept a note of every meeting where a conversation stalled because people were clearly picturing different things. By Friday the list was short and revealing:
- In production — serving live decisions, or merely receiving live data?
- Accuracy — on the original test set, on last month’s labelled data, or the share of flagged cases later confirmed?
- Drift — a change in incoming feature distributions, or a fall in a performance metric?
- Retrained — a new model version with new data, or the same pipeline re-run on a schedule?
- Incident — any model alert, or only events that affected customers?
Her advice to other leads is simple: do not guess which terms matter. Listen for the ones that repeatedly derail a meeting. Those are the ones costing you time.
Step 2: Ask everyone to write their own meaning first
Before any discussion, Ana sent each team member a small form: one line per term, “what do you mean when you say this?” She asked them not to look anything up. The answers came back wonderfully inconsistent. Two engineers had defined drift almost identically; a third had written something that described a data pipeline failure. A data scientist had defined “in production” with a careful caveat about shadow mode that nobody else had considered.
Collecting private answers first matters. In an open meeting, people tend to converge on whatever the most senior person says, and the useful disagreements disappear. On paper, the real range of meanings is visible.
Step 3: Meet once and settle on one sentence per term
With the answers printed side by side, the team met for an hour. The rule was that every term would leave the room with one sentence, a way to check it, and nothing more. For example:
- In production: the model’s output directly drives a customer-facing decision. Shadow or canary scoring is called “pre-production.”
- Accuracy (reporting): precision on confirmed cases from the previous calendar month, as calculated by the evaluation job.
- Drift: a statistically significant shift in an input feature’s distribution compared with the training window. A fall in performance is called “degradation,” separately.
- Retrained: a new model version trained on refreshed data and registered with a new version number.
Splitting overloaded words was the biggest win. Once “drift” and “degradation” had separate names, a long-running argument about whether the model was drifting simply evaporated. It had been drifting slightly and degrading not at all.
If your team is still mapping out the stages of its machine learning lifecycle, our staged MLOps learning plan for scientists and engineers helps place each of these terms in context.
Step 4: Give every definition a single owner
Ana’s first draft had no owners, and within a week someone had quietly edited the accuracy entry to suit a presentation. Nobody objected because nobody felt responsible. The fix was to assign one owner per term:
- The lead data scientist owns accuracy and degradation.
- The platform lead owns in production and pre-production.
- The ML engineer responsible for monitoring owns drift.
- The on-call lead owns incident.
Owners do not need to defend their wording against every challenge. They need to approve any change, date it and tell the team. Ownership turns the glossary from a suggestion into a reference.
Step 5: Align operational terms with your reliability practice
The incident definition turned out to be the hardest, because it touched the wider engineering organisation. The site reliability team already had severity levels and an incident process, and the ML team’s casual use of “incident” for any model alert clashed with it. Rather than invent a parallel scheme, Ana’s team adopted the existing SRE definitions and added a single clarification for model-specific events. Our article on improving incident handling with SRE habits and automation describes the kind of process worth aligning with.
The lesson: where a broader definition already exists and works, borrow it. Your glossary should reduce the number of dialects, not add one more.
Step 6: Put the glossary where the words are used
A glossary in a wiki nobody opens is a glossary nobody follows. Ana’s team placed it in three spots:
- At the top of the model repository’s README, so every engineer sees it when cloning.
- As tooltips beside metric names on the team’s monitoring page, so the meaning of “accuracy” is one hover away.
- On the first slide of the quarterly review template, so business stakeholders read the same definitions the team uses.
That last placement had the most visible effect. At the next review, when the business owner asked about accuracy, the answer came with a footnote she could read for herself. The conversation moved straight to what to do about it.
Step 7: Change definitions out loud, never quietly
Three months later the fraud team switched to a new labelling partner, and the definition of a confirmed case had to change. The owner updated the accuracy entry, wrote the date and reason beneath it, and posted a short message to the team channel. The next quarterly report included one line noting that figures before and after the change were calculated differently. Nobody was surprised, nobody accused anyone of moving the goalposts, and the trend line was read with the right caution.
Clear wording also matters for service reliability more broadly. The team’s agreed meaning of “pre-production” made it easier to explain why certain alerts did not page anyone at night, which in turn supported the kind of steady availability described in our piece on how reliability engineering cuts downtime for users.
What Ana would tell another team lead
Looking back, she says the glossary was the least technical improvement the team made all year and possibly the most useful. A few lessons stood out:
- Keep it short. Seven entries cover nearly every recurring confusion. Resist the urge to define everything.
- Prefer splitting to arguing. When two meanings are both legitimate, give them two names.
- Include a check. A definition you cannot verify against a job, a record or a dashboard will drift back into opinion.
- Revisit after big changes. New data sources, new partners or new platforms are when meanings slip.
Where to learn the wider discipline
Ana’s experience mirrors a practice that data teams treat as foundational: agreed definitions underpin data quality and governance, which in turn underpin every model built on that data. Teams wanting a structured route into those practices can explore DataOpsSchool’s DataOps courses and consulting for data and ML teams, which cover quality and governance within a wider operating model for data work.
Your turn
You can follow the same path in a fortnight. Spend a week noticing the words that stall meetings. Collect private definitions. Meet once to agree a single sentence and a check for each. Assign owners. Place the glossary where people already look. And when a meaning must change, change it in public. Ana’s quarterly reviews are now shorter, calmer and far more focused on decisions. The model did not change at all. The words around it did.