Learn AI · Level 2 · How AI works · step 2 of 16 · word 4 of 6
What is gradient descent?
Gradient descent is how a model learns. It measures how wrong it is, then nudges its settings a little in the direction that makes it less wrong, millions of times over.
In plain English
Picture walking down a foggy hill to reach the bottom. You can’t see far, so you feel which way the ground slopes and take a small step downhill. Repeat, and you get there.
For a model, the “hill” is how wrong its answers are. Each step adjusts its parameters to make the error a little smaller.
Backpropagation
A neural network can have billions of parameters. Backpropagation is the method that works out which way to nudge each one. It became popular after a 1986 paper by Rumelhart, Hinton and Williams, and almost all of today’s AI is trained with it.
Watch out
Training this way takes huge amounts of computing power, which is why big models cost so much to build. See how AI chatbots work.
The score that gradient descent pushes down is called the loss function.