What is a diffusion model?

In plain English

Imagine a photo slowly covered in static, like an old TV, until nothing is left but grey fuzz. A diffusion model is trained to run that process backwards. During training it is shown millions of images with increasing amounts of noise added, and it learns to predict and remove a little of the noise at each step.

To make a new image, it starts from pure random noise and removes noise again and again, maybe 20 to 50 times. Your text prompt steers each step, so the shapes that emerge from the fuzz become “a red bicycle leaning on a blue door” rather than anything else.

Why it matters

Diffusion models are why text-to-image tools took off in 2022, with OpenAI’s DALL·E 2 and the openly released Stable Diffusion. OpenAI has described its Sora video model as a diffusion model too, and the same approach is used for audio. They are the “image” half of generative AI, alongside language models for text.

An example

You type “a watercolour of a lighthouse at sunset” into an image generator. Behind the scenes it starts from a square of random speckles. After the first steps you would see only vague light and dark patches; halfway through, a tall shape and a glowing sky; by the last step, brushstrokes and colour, matching your words.

What to watch out for

Because they learn from huge collections of images from the web, diffusion models raise hard questions about artists’ copyright, and several lawsuits are under way. They also make convincing fakes easy to create, so a realistic picture is no longer proof that something happened: see deepfakes and our guide to spotting AI fakes.

Watch it happen, step by step, with the Diffusion Explainer in our AI Playground.