A model that works in a notebook and a system a business can depend on are separated by four pieces of engineering. Teams skip all four, then wonder why the pilot never shipped.
The most common shape of a failed AI project is not a model that doesn't work. It is a model that works beautifully, once, on a laptop, and is never seen again.
This is not a modelling failure. It is a systems-engineering failure, and it is remarkably consistent. Four pieces of work sit between a notebook and something a business can depend on. Teams skip them in the same order, for the same reasons, and the pilot stalls in the same place.
The first version of almost every model is trained on a dataset that exists in exactly one place: a folder on someone's machine, assembled through a sequence of steps that were not written down. It was joined, filtered, deduplicated and patched by hand, and the person who did it remembers most of it.
This is fine for finding out whether the idea has legs. It is fatal the moment anyone asks the obvious question: can you retrain it?
The fix is unglamorous and it is the highest-leverage hour in the whole project. Turn the data preparation into code that runs start to finish, from a source you can point at, and produces the same dataset twice. Not a pipeline framework. Not orchestration. A script that runs.
Skip this and every subsequent improvement is unverifiable, because you can never tell whether the model got better or the data quietly changed.
In a notebook these are the same thing: you fit, you predict, you look at the output. In production they are different systems with different constraints, different failure modes and usually different hardware.
The specific trap is feature computation. Training features get computed in bulk, over historical data, with the luxury of hindsight. Serving features get computed one at a time, in milliseconds, with only what is known right now. When those two paths are written separately — which is what happens when nobody has drawn the boundary — they drift apart, and the model performs worse in production than in testing for reasons nobody can locate.
Draw the boundary deliberately. One definition of each feature, used by both paths. This single decision prevents more production incidents than any amount of model tuning.
Most notebook evaluation is a number on a test split, and most of the time that number is optimistic for a reason nobody has looked for: leakage, a split that ignores time, or a metric that rewards the majority class.
The question to ask is not “is the score good” but “what would this score look like if the model were useless?” If you cannot answer that, the score means nothing. A fraud model with 99.4% accuracy on a dataset that is 99.4% legitimate has learned to say no.
Then go further: evaluate on the slices you would be embarrassed to get wrong. Aggregate accuracy hides exactly the failures that end up in front of a regulator — the small segment where performance collapses, invisible in the mean.
An evaluation you would defend in a hostile room is a different artefact from one that confirms the model works. Build the first kind.
The fourth step is skipped most often, because by the time you reach it the project feels finished. It is not. A deployed model is a claim about the world, and the world moves.
Monitor three things, in increasing order of difficulty. Whether the system is up. Whether the inputs still look like the inputs it was trained on. And whether the predictions were right — which usually requires waiting for reality to arrive and joining it back to what you predicted, and is the piece almost nobody builds.
The last one is the only one that tells you the truth. Without it you will find out that the model degraded when a person outside your team notices something is wrong.
And build it for a reader. A dashboard nobody opens is not monitoring; it is a more expensive way of not knowing.
None of this is difficult, and that is the interesting part. It gets skipped because it is invisible. Nobody demos a reproducible dataset. There is no slide for a feature boundary. The work that makes the difference between a pilot and a system produces nothing to show a stakeholder, so it loses to work that does — right up until the moment it is the only thing that matters.
This is the gap our engineering cohorts are built around. Not because the modelling is unimportant, but because the modelling is the part most people can already do.
