Term explained
What is AI bias?
AI bias is when an AI system treats some groups of people unfairly, usually because it learned patterns from data that reflected past unfairness.
In plain English
AI systems learn from examples. If the examples are unbalanced or reflect unfair decisions people made in the past, the AI learns those patterns too, and then repeats them, quickly and at scale. That is AI bias.
It can creep in in several ways: the data may contain far more examples of some groups than others, the past decisions it learns from may have been unfair, or the thing being measured may be a poor stand-in for what really matters.
Why it matters
AI is used to help make decisions about people: who gets a job interview, a loan, extra checks at a border, or a police investigation. A biased system can treat thousands of people unfairly before anyone notices, and because it is a computer, its decisions can look more neutral than they are.
An example
Amazon built an experimental tool to rate job applicants’ CVs. Trained on ten years of past CVs, mostly from men, it learned to mark down CVs that included the word “women’s”, as in “women’s chess club captain”. Amazon scrapped it. More cases are on our AI fails page.
What to watch out for
Ask whether an AI system that makes decisions about people has been tested for how it treats different groups, and whether a person reviews its decisions. Chatbots can show bias too, for example in the assumptions they make about someone’s job or gender. Notice when an answer leans on a stereotype, and push back.