What is red teaming in AI?

In plain English

The name comes from military exercises, where a “red team” plays the enemy to test a plan. In AI, red teamers try everything a bad actor or a careless user might: clever wording to get around the rules (jailbreaks), hidden instructions in documents (prompt injection), requests for dangerous information, and long conversations that slowly push the model off course. Whatever gets through is reported, fixed and turned into tests.

Red teams can be the lab’s own staff, hired outside experts (for example in cyber security or biology), government institutes such as the UK’s AI Security Institute, or members of the public in organised challenges.

Why it matters

Normal testing checks that an AI does what it should. Red teaming checks what it does when someone is trying to make it misbehave, which is exactly what will happen once it’s public. Leading AI labs red team their major models before release and describe the results in the safety reports they publish alongside them.

An example

Before launching a customer-service chatbot, a bank asks a small team to spend a week attacking it. They find that pretending to be a “bank employee doing a test” makes it reveal internal policy notes, and that a polite request in another language gets around its rules on investment advice. Both are fixed, and the attacks are added to the bank’s evals so they’re checked after every change.

What to watch out for

Red teaming finds problems; it can’t prove there are none left. New tricks keep appearing after launch, so it works best alongside guardrails, monitoring and a way for users to report problems. You can try a playful version yourself with Gandalf, a game where you trick an AI into revealing a password.

Read more in AI safety and the rules.