Here’s a situation almost every data scientist has lived through. Three weeks ago you trained a model that scored noticeably better than everything else. Today your manager asks you to reproduce it for a demo. You open your notebooks and find six files with names like final_v2_really_final, two different feature sets and no idea which learning rate you used. Sound familiar? That’s the problem an ML experiment log solves. In plain words, it’s a dated, running record of what you tried while building a model, what happened, and what you decided to do next.
Let’s unpack what that means, why it matters so much in machine learning work, and how to keep one without turning it into a chore.
What is an ML experiment log, really?
Think of it like a lab notebook, the kind chemistry students are told to keep, but for model building. Each time you run something meaningful, you jot down a few lines:
- The date and a short name for the run.
- What you changed compared with last time (data, features, settings, code).
- What you expected to happen.
- What actually happened, with the scores you looked at.
- What you’ll try next, or why you’re stopping this line of work.
That’s it. No essays. A good entry is often five or six lines long.
Doesn’t experiment tracking software already do this?
Partly, and those tools are genuinely useful. They capture parameters, metrics and model files automatically, which is a huge help. But they don’t capture your thinking. They’ll tell you run 147 used a learning rate of 0.001. They won’t tell you that you tried it because you suspected the model was overshooting, or that you noticed a weird spike in validation loss you meant to come back to.
The written log fills in the “why” and the “hmm, that’s odd” moments. In practice the two work best together. Many people simply paste the run ID from their tracking tool into the log entry, so the numbers and the reasoning point to each other.
Why it matters more in ML than in most software work
Ordinary software tends to behave the same way each time you run it. Machine learning doesn’t. Results shift with random seeds, data samples, library versions and small preprocessing tweaks. You can easily run dozens of experiments in a week, and they start to blur together after just a few days.
That’s why reproducibility is such a big theme in the field. When teams talk about the importance of operational discipline around models, as in this piece on why MLOps practices matter for getting machine learning right, a lot of it comes back to being able to answer one simple question: how exactly did we get this result?
A quick example
Here’s what a couple of days in a churn-prediction project might look like:
- Mon 2 Nov, run “baseline-gbm”: Gradient boosting with default settings on last quarter’s data. Validation AUC looks decent. Next: add the support-ticket count feature.
- Mon 2 Nov, run “tickets-added”: Small gain. Feature importance shows tickets near the top. Next: try ticket counts over the last 30 days only.
- Tue 3 Nov, run “tickets-30d”: Bigger gain, suspiciously big. Checked: 30-day window overlaps the churn label date. That’s leakage. Dropped it.
- Tue 3 Nov, note: Lesson: always check feature windows against label dates. Adding to my checklist at the top of the log.
Notice entry three. Without a log, that “great result” might have made it into a demo, and the leakage might only have been spotted after deployment. The log made the reasoning visible, and the lesson got written down so it doesn’t have to be re-learned on the next project.
How the log helps your team, not just you
Machine learning projects are full of handovers. A data scientist explores, an ML engineer productionises, someone else monitors the model months later. Each handover loses context unless it’s written down somewhere.
A good experiment log answers the questions the next person will have: which approaches were already tried and why they were dropped, which data issues were discovered, what assumptions the final model depends on. Teams building a long-term plan for their ML work, like the stages laid out in this roadmap for data scientists and engineers adopting MLOps, often find that the humble log is what makes the early stages smooth.
It also speeds things up. When you can see that a certain feature idea was tested and failed in March, nobody wastes a week trying it again in October. That’s a real effect on delivery time, which ties into the broader point made in this article about how mature ML operations shorten the path from model to production.
Tips to make it stick
- Write as you go. Add the entry when the run finishes, not at the end of the day.
- Keep one log per project. A single Markdown file in the repo works well and stays with the code.
- Record failures. Dead ends are the most valuable entries, because they stop others repeating them.
- Use plain language. Write so a teammate could follow it without asking you.
- Review weekly. Skim the week’s entries on Friday and pull out any lessons worth keeping at the top.
- Leave out secrets. No credentials or customer data in the log, ever.
Where to see real-world examples
If you’d like to see how practitioners structure their notes, it helps to read a few. AI Universe, which writes about machine learning, data science and AI governance, has a Stories section where people working in the field share their daily logs. Skimming a handful will give you a feel for how short and specific good entries can be.
What about notebooks? Aren’t they a log already?
Good question. Jupyter-style notebooks mix code, output and notes, so they feel like a record. The catch is that people rerun cells, overwrite outputs and delete the experiments that didn’t work. A notebook shows the latest state, not the journey. Your log is the journey. Some people add a short markdown cell at the top of each notebook pointing to the matching log entry, which keeps the two connected without duplicating effort.
The short version
An ML experiment log is a few dated lines per run: what you changed, what you expected, what happened, what’s next. It sits alongside your tracking tools and captures the reasoning they can’t. It protects you from losing that great result, from repeating dead ends and from painful handovers. And it costs maybe five minutes a day. Start with your next run.