How a machine learning model is built: from data to drift

The 8 steps, from idea to a model that keeps working

The model itself is built in step 5. Everything around it is what makes it work in real life.

1. 1

   Frame the problem

   Decide what the model should predict, and how you will know it is good enough. Is the answer a category or a number? What does a mistake cost?

   In our exampleThe bank wants to flag card payments that are likely to be fraud, a yes-or-no answer. Missing fraud costs money; blocking a genuine payment annoys a customer.

   What goes wrong: Solving the wrong problem well. A model that predicts something easy to measure, instead of what the business needs, can score high and help nobody.

   Product managers and business owners, with data scientistsClassification vs regressionPredictive AI

2. 2

   Collect the data

   Gather past examples that look like the cases the model will face: as many, as varied and as recent as you can, and only data you are allowed to use.

   In our exampleFive years of card payments: amount, shop, country, time of day and the device used, for millions of transactions.

   What goes wrong: Data that leaves people out. If the past data mostly covers one kind of customer, the model works worst for everyone else.

   Data engineersTraining dataSynthetic dataAI bias

3. 3

   Label it

   Tag each example with the right answer, so the model has an answer key to learn from. Often this is slow work done by people.

   In our exampleEach past payment is marked "fraud" or "genuine", mostly from customer complaints and chargebacks the bank has already checked.

   What goes wrong: Wrong labels. The model learns the mistakes, and if the test data has wrong labels too, the scores are wrong as well.

   • An MIT-led study found errors in at least 3.3% of labels across ten widely used test datasets, and at least 6% in the ImageNet validation set. Source
   • In June 2025, Meta invested in the labelling company Scale AI in a deal that valued it at over $29 billion. Source

   Data labellers, often through specialist firms, and experts for hard casesData labellingSupervised learning

4. 4

   Split it

   Lock away part of the data so the model never sees it while learning. It becomes the final exam.

   In our exampleThe model learns from the first four years of payments. The fifth year is kept back to test it, just as it will face payments it has never seen.

   What goes wrong: Leaks. If test examples slip into the training data, the model has seen the exam answers, and its scores mean little.

   Data scientistsTrain-test splitOverfitting

5. 5

   Train it

   The model guesses, a loss function scores how wrong it was, and gradient descent adjusts it to do a little better, over and over. People pick the settings (hyperparameters), such as how fast it learns and for how long.

   In our exampleThe model sees millions of labelled payments and slowly learns which mixes of amount, place, time and device tend to mean fraud.

   What goes wrong: Memorising instead of learning. Train too long on too little data and the model learns the quirks of its examples, not the real pattern.

   Data scientists and machine learning engineersLoss functionGradient descentParametersNeural network

6. 6

   Test it

   Run the model on the locked-away data and measure the mistakes that matter, not just overall accuracy. Then decide whether it is good enough to use.

   In our exampleOn the held-back year, the team checks how much fraud the model catches (recall) and how many of its alarms are real (precision), and picks an alarm setting.

   What goes wrong: Trusting one headline number. When fraud is rare, a model can be 99% accurate and catch nothing. Try it in the demo below.

   Data scientists, with product and risk teams signing offPrecision and recallEvals (AI evaluations)Benchmark

7. 7

   Put it to work

   Package the model so a live system can ask it for answers in a fraction of a second, release it carefully and keep the old version ready to roll back.

   In our exampleEvery time a card is used, the payment system asks the model for a fraud score before approving it. The new model first runs alongside the old one on a small share of payments.

   What goes wrong: A model that worked in the lab but not in real life: too slow, fed slightly different data, or with nobody owning it once it is live.

   • Google engineers warned in 2015 that real-world machine learning systems carry large ongoing maintenance costs, from data dependencies to changes in the outside world. Source

   Machine learning engineers and MLOps or platform engineersMLOpsInferenceCI/CD

8. 8

   Watch it and retrain

   Keep checking the live model: is the data coming in changing, and are its answers getting worse? When they are, retrain on fresh data and start the loop again.

   In our exampleFraudsters change tactics and shoppers change habits. The bank tracks how much fraud gets through each week and retrains the model every few months, or sooner if the numbers slip.

   What goes wrong: Drift. Nothing in the code breaks, so nobody notices the model getting slowly worse until it is costly.

   • In spring 2020, the pandemic changed shopping so fast that models for stock, fraud and marketing, trained on normal behaviour, started making bad calls and people had to step in. Source
   • In 2021, Zillow closed its home-buying business after it bought homes for more than it could sell them for. It wrote down about $304 million of homes in one quarter, and its CEO said home prices had proved far harder to forecast than expected. Source

   MLOps engineers and data scientistsModel driftMLOpsHuman-in-the-loop

Then back to step 2: fresh data, new labels, retrain and test again. A model is never finished, and keeping this loop running is what MLOps is for.

Try it: catch the fraud

Here are 1,000 card payments, and 10 of them are fraud. Pick how strict the model is before it raises an alarm, and watch what happens. (Illustrative numbers.)

Notice that accuracy barely moves, and is highest when the model is least useful. That's why teams judge models on precision and recall, and pick the setting from the cost of each kind of mistake.

What changes for chatbots (LLMOps)

Apps built on large language models follow the same loop, but most of the steps look different.