Learn AI · Level 1 · Foundations · step 11 of 18
AI hype vs reality: 8 big claims, checked
AI news is full of bold claims. Some are already true, some are half true, and some are predictions dressed up as facts. We took eight you hear all the time, checked what has actually been shown, and gave each one a label. Every figure links to its source.
- 29%more breast cancers found when AI helped doctors read scans, in a trial of 100,000+ womenSource: lunduniversity.lu.se
- 19%slower: experienced developers using AI tools on their own projects, in a 2025 trialSource: metr.org
- ~30%of ordinary office tasks finished by the best AI agent in a pretend companySource: arxiv.org
Our four labels
- DemonstratedShown in real use or a large, independent study.
- Early-stageReal but patchy: works in places, not yet reliably.
- Still speculativeA forecast or a promise. No one can show it yet.
- Not supportedThe evidence points the other way.
DemonstratedChecked Oct 2026
“AI can spot cancers that doctors miss.”
- What's been shown
- In a Swedish trial with more than 100,000 women, mammogram screening supported by AI found 29% more cancers than two doctors reading alone, with no rise in false alarms, and cut doctors' reading work by 44%.
- What's missing
- This is one narrow task, done with doctors still in charge. The trial's main result, how many cancers appear between screenings, comes after two years of follow-up.
- Our verdict
- True for screening scans, as a second pair of eyes. It is not proof that AI diagnoses better than doctors in general.
DemonstratedChecked Oct 2026
“Cars that drive themselves are already here.”
- What's been shown
- Waymo runs paid rides with no one in the driver's seat in 14 US cities, more than 500,000 rides a week. Over 25 million miles, it had 88% fewer property-damage claims and 92% fewer injury claims than human drivers, in a study with the insurer Swiss Re.
- What's missing
- It works only in mapped areas of chosen cities, and much of the rest of the industry still uses safety drivers or remote monitors. A car you buy that drives itself anywhere does not exist yet.
- Our verdict
- Real, and safer than people in the places it runs. "Anywhere, in your own car" is still a long way off.
AI and robots →Waymo reaches 14 cities (Sept 2026)Waymo and Swiss Re safety study
Early-stageChecked Oct 2026
“AI will replace entire software teams.”
- What's been shown
- AI writes a real share of code: Google said in 2024 that more than a quarter of its new code is generated by AI, then reviewed and accepted by engineers.
- What's missing
- In a 2025 trial by the research group METR, experienced developers working on their own projects were 19% slower when allowed AI tools, while believing they were 20% faster. Someone still has to decide what to build, check the code and take responsibility for it.
- Our verdict
- A useful assistant that changes the job. Whole teams run by AI is not what the evidence shows.
AI tools for coding →Google Q3 2024 earnings remarksMETR developer study (July 2025)
Early-stageChecked Oct 2026
“AI agents can do your job for you.”
- What's been shown
- Agents can now browse websites, fill in forms, write code and use apps on their own, and they handle short, well-defined tasks well.
- What's missing
- In a Carnegie Mellon test that set agents loose in a pretend software company, the best one finished only about 30% of ordinary office tasks. Gartner expects more than 40% of company agent projects to be cancelled by the end of 2027.
- Our verdict
- Good for small, checkable chores. Handing over a whole job is not realistic yet.
AI agents, explained →TheAgentCompany study (CMU)Gartner prediction (June 2025)
Still speculativeChecked Oct 2026
“AI is about to wipe out most jobs.”
- What's been shown
- There are early signs at the bottom of the ladder: a Stanford study found employment for 22 to 25-year-olds in the most AI-exposed jobs, such as customer service and software, fell 13% compared with other workers.
- What's missing
- Across the whole US job market, Yale researchers found no clear disruption in the years since ChatGPT, and older workers in the same jobs held steady or grew. Past technologies took decades to reshape work.
- Our verdict
- Entry-level jobs are worth watching. Mass job loss is a prediction, not something that has happened.
Before AI vs after AI →Yale Budget Lab: AI and the labour marketStanford "Canaries" study, via The Register
Still speculativeChecked Oct 2026
“Human-level AI is only a couple of years away.”
- What's been shown
- Some lab leaders say so: Anthropic's chief executive wrote that very powerful AI could arrive as early as 2026. Models do keep improving fast on tests.
- What's missing
- Researchers disagree widely. In a survey of 2,778 AI researchers, the middle answer for when machines could do every task better and more cheaply than people was 2047. Today's models still make basic mistakes no human expert would.
- Our verdict
- A forecast, with honest experts decades apart. Treat any exact date with caution.
The AI ladder →Survey of 2,778 AI researchers (2024)Dario Amodei, "Machines of Loving Grace"
Not supportedChecked Oct 2026
“The latest chatbots no longer make things up.”
- What's been shown
- Newer models make fewer mistakes, and answers that search the web and show sources are easier to check.
- What's missing
- In a 2025 study by 22 public broadcasters, 45% of AI assistants' answers about the news had at least one serious problem, such as wrong facts or sources that did not say what was claimed.
- Our verdict
- Better, but not fixed. Check names, numbers and quotes before you rely on them.
Not supportedChecked Oct 2026
“Every AI question uses a bottle of water.”
- What's been shown
- Data centres do use a lot of water and power in total, and the AI boom is adding to it quickly.
- What's missing
- Google measured a typical Gemini text question at about 0.26 ml of water, around five drops, and 0.24 Wh of electricity. The "bottle" figure came from an earlier estimate for 10 to 50 answers that also counted water used at power stations, for older models.
- Our verdict
- One text question is tiny. The real issue is the total, and video and image generation use far more.
How much energy does one AI question use? →Google: energy and water per AI prompt"Making AI Less Thirsty" (2023 estimate)
Five questions to ask about any AI claim
Next time a headline promises the moon, run it through these.
- Who is saying it? A company selling the product, or someone who tested it independently?
- Was it tested, or demoed? A demo shows the best case. A study shows the usual case.
- Shown for what, exactly? "Better at one scan type" quietly becomes "better than doctors".
- Is it a result or a date? "By 2027" is a forecast. Ask what has happened so far.
- Who still checks the work? Most real successes keep a person in charge.
How we put this guide together
We chose claims that come up again and again in news, social media and sales pitches. For each one we looked for the strongest independent evidence we could find: randomised trials, large studies and official figures, and we name the source whenever a number comes from the company making the claim. The labels are our judgement of that evidence, not a prediction. Claims about AI age quickly, so every card shows when we last checked it, and we review this page every quarter. Think we’ve got one wrong, or want a claim checked? Tell us.
Learn AI path · Level 1 · Foundations · step 11 of 18
Done? Next stepAI fails: when AI got it badly wrong (and what to learn) →See all of level 1