The graveyard of applied ML is full of prototypes that "worked."
They worked in the sense that a notebook produced a number. They failed in the sense that nobody could say, six weeks later, whether the number was still true.
At Iteration Matrix the visible symptom was reporting error: about one record in four. That is not an AUC problem. That is a trust problem. People stop opening the dashboard. The model becomes folklore.
What I mean by MLOps
I do not mean a vendor. I mean a lifecycle you can run without me in the room:
- A data contract.
- A training path that is not a laptop.
- A registry, so versions are objects.
- A deploy that can be rolled back.
- Monitoring on data and on decisions.
- A retrain that is scheduled, not inspired.
MLflow, Docker, and CI/CD showed up because they were sufficient. The stack is not the point. The refusal to let a model be a one-off is the point.
Monitor the decision
Accuracy on a holdout is a starting condition. The thing that changed the error rate was watching the decision: did this supplier match, was this dollar attributed, did this recovery fire, and can a human replay why?
When you instrument that, you find the boring bugs — joins, late data, silent schema changes — that a confusion matrix will never mention.
I wrote the case as ML Systems That Stay Up. If you are a student, this is the part of the job that tutorials skip and employers quietly select for.