Term explained
What is an AI benchmark?
A benchmark is a standard test used to measure and compare AI models, such as a set of exam questions, coding tasks or puzzles, scored the same way for every model.
In plain English
When an AI company launches a new model, it usually publishes a table of scores: so many percent on a maths test, so many on a coding test, and so on. Each of those tests is a benchmark. A benchmark is like a standard exam that every model sits, so their results can be compared fairly.
There are benchmarks for almost everything: school and university exam questions, real programming bugs to fix, science questions written by experts, and puzzles designed to be easy for people but hard for AI. When people say a model “tops the leaderboards”, they mean it scored highest on popular benchmarks.
Why it matters
Benchmarks are the main way to tell whether AI is improving and which model is best at what. They also show how fast things are moving: tests that were meant to last years are often “solved” within months, so researchers keep writing harder ones, such as Humanity’s Last Exam, a set of very difficult questions from experts across many fields.
An example
Suppose a new coding model fixes 70% of the problems in SWE-bench, a benchmark made of real bugs from public software projects, while the previous version fixed 50%. That would suggest it is better at practical programming work, not just at writing short snippets.
What to watch out for
A high score is not the same as being useful for your task. Models can be tuned to do well on popular tests, and benchmark questions sometimes leak into the data a model is trained on, so it has effectively seen the answers. Companies also choose which scores to show. Treat benchmarks as a rough guide, and try a model on your own real work before deciding.