What is interpretability in AI?

In plain English

A modern AI model is a neural network with billions of numbers (its parameters) that nobody wrote by hand: they were learned during training. We can see every number, but that doesn’t tell us what they mean, much as a brain scan doesn’t show you someone’s thoughts. Interpretability researchers build tools to look inside and find which parts of a model represent which ideas, such as a particular city, a programming error or the concept of a lie, and how those parts combine to produce an answer.

A related, older term is “explainable AI”: methods that explain a single decision, such as which words in a loan application mattered most.

Why it matters

Today, we mostly judge AI by testing its behaviour. But a model could behave well in tests and differently elsewhere, and testing can’t cover every situation. If we could read what a model is “thinking”, we could spot hidden problems, such as bias, deception or a planted backdoor, directly. That’s why AI labs, including Anthropic, Google DeepMind and OpenAI, have interpretability teams, and why it’s often named as one of the most important parts of AI alignment.

An example

In 2024, Anthropic researchers found a feature inside their Claude model that lit up for the Golden Gate Bridge, in text and images. When they turned that feature up, the model began bringing the bridge into almost every answer, even describing itself as the bridge. It showed that specific ideas can be found inside a model and changed on purpose.

What to watch out for

The field is young. Researchers can explain small pieces of how models work, not the whole thing, and the biggest models are getting more complex faster than the tools improve. Treat claims that a model is “fully understood” or “fully explainable” with caution.

Read more about how labs keep models safe in AI safety and the rules.