What is a transformer in AI?

In plain English

A transformer is a particular way of building a neural network, and it is the design behind almost every modern AI model that works with language. Older language systems read text one word at a time, like someone following a sentence with their finger, and tended to forget the start of a long passage by the time they reached the end. A transformer takes in the whole passage at once and uses a technique called attention to work out which words matter to which. That lets it keep track of meaning across long documents.

The design was introduced by Google researchers in a 2017 paper called “Attention Is All You Need”. It turned out to work for far more than translation, its original purpose.

Why it matters

Transformers are why AI suddenly got good at language. The “T” in GPT stands for transformer (Generative Pre-trained Transformer), and the same design now powers image, audio and video models too. Because transformers process text in parallel rather than word by word, they make full use of GPUs, which is what made training today’s huge large language models possible.

An example

Take the sentence “The trophy didn’t fit in the suitcase because it was too small.” To know what “it” means, you need to connect “it” with “suitcase”, not “trophy”. A transformer looks at all the words together and learns these links from billions of examples during pre-training.

What to watch out for

A transformer is a design, not a mind. It is very good at finding patterns in text, but it has no built-in way to check facts, which is one reason models built on it can still make things up. And the attention step gets expensive as text grows, which is why every model has a context window limit.

To see one working step by step, try the Transformer Explainer in our AI Playground.