Learn AI · Level 2 · How AI works · step 6 of 19 · word 1 of 4
What is a data pipeline?
A data pipeline is the automated route data takes from where it is created to where it is used, being checked and cleaned along the way. AI models are fed by them.
In plain English
Data rarely starts where it is needed. It is spread across apps, websites, sensors and spreadsheets. A data pipeline is a set of automatic steps that collects it, fixes the mess (missing values, duplicates, different date formats) and delivers it to a store such as a data warehouse or data lake. Most pipelines follow a recipe called ETL.
An example
Every night, a bank’s pipeline gathers the day’s card payments from its systems, removes test transactions, adds the customer’s country and saves the result. Its fraud model learns from, and is checked against, that data.
Watch out
Pipelines can break quietly. If one source changes a column name, a model can keep running on wrong or missing data for weeks. More in how a machine learning model is built.
Learn AI path · Level 2 · How AI works · step 6 of 19
Next wordETL (extract, transform, load) →See all of level 2