What are scaling laws in AI?

In plain English

In 2020, researchers at OpenAI published a paper showing something surprising: a language model’s performance improves smoothly and predictably as you scale up three things together: the size of the model (its parameters), the amount of training data, and the computing power used to train it. Plot the results on a chart and they fall along a straight line. Double the resources and you can forecast, before training, roughly how much better the model will be.

In 2022, DeepMind refined the recipe with a model called Chinchilla. It found many big models had been trained on too little data for their size, and that model size and data should grow together.

Why it matters

Scaling laws are the economic logic behind modern AI. If spending ten times more reliably buys a better model, companies have a reason to build enormous data centres, buy hundreds of thousands of chips and sign deals for whole power stations. They are also why some people expect continued rapid progress towards AGI: if the lines keep going, the models keep getting better.

An example

Before spending hundreds of millions of dollars training a big model, a lab first trains a series of small ones, measures how they improve as they grow, and uses the scaling-law trend to predict how good the big one will be, and how much data and computing power it needs.

What to watch out for

Scaling laws are patterns seen so far, not laws of nature. They predict a model’s score on a technical measure (how well it predicts text), which doesn’t translate neatly into useful skills. There are open questions about running out of good human-written data, the cost and energy of each step up, and whether progress now comes more from new techniques, such as reasoning models that “think” longer, than from size alone.