What is pre-training in AI?

In plain English

A chatbot like ChatGPT is built in stages, and pre-training comes first. The model is shown an enormous amount of text (books, websites, code, articles) and given one simple task over and over: guess the next word. Each time it guesses wrong, its settings (its parameters) are nudged so it would guess a little better next time. After trillions of these guesses, it has absorbed grammar, facts, styles of writing and some ability to reason, simply because all of that helps predict what comes next.

The result is a “base model”, also called a foundation model: knowledgeable, but not yet a helpful assistant. It might answer a question with another question, because that’s what often comes next on the web.

Why it matters

Pre-training is where most of a model’s knowledge, and most of its cost, comes from. It can take months on thousands of GPUs in a data centre. It is also where the “P” in GPT comes from (Generative Pre-trained Transformer). Everything that follows, such as fine-tuning and RLHF, is a much smaller step that shapes the base model into a useful assistant.

An example

Think of a medical student. Pre-training is the years of reading every textbook in the library: broad knowledge, no bedside manner. Fine-tuning and RLHF are the residency, where they learn how to talk to patients and what a good answer looks like.

What to watch out for

A model only knows what was in its pre-training data, up to the date it was collected (its “knowledge cutoff”). It also inherits the data’s gaps, errors and biases. Who gets to use which text for pre-training is one of the biggest open questions in AI, with copyright lawsuits under way in several countries.