# How to read an AI claim

Explainer · Updated 2026-09-07
Canonical: https://aiforthedummies.com/articles/how-to-read-ai-claims/

AI marketing is fluent for the same reason the models are fluent: it is cheap to produce a finished-looking sentence. Before you repeat a number — “beats doctors,” “#1 on the leaderboard,” “cuts costs in half” — ask what object was measured, on what data, against what alternative, and who paid for the test.

**This desk will not fill in a missing score.** If a press release omits the dataset, the date, or the comparison system, we will not invent them. We also will not paste third-party leaderboard tables into these pages. Leaderboards change; they are not a substitute for a named evaluation card.

## Five questions that usually suffice

1. **Model or product?** A score on a raw model is not a score on the consumer app with tools and filters. See [models versus products](https://aiforthedummies.com/articles/models-vs-products/).
2. **What task?** “Reasoning” and “intelligence” are slogans until someone names a test: multiple-choice questions, coding puzzles, translation pairs, a human preference vote. Different tasks disagree.
3. **What data?** If the test items leaked into training, the number is closer to memorization than to skill. If the data is private, you cannot reproduce the claim.
4. **Compared with what?** Beating last year’s own model is not the same as beating a competent person, a search engine, or a cheaper model. “Human-level” needs a definition of the human and the time allowed.
5. **Who ran it, and can you read the method?** A vendor blog is a claim. A paper with prompts, seeds, and error bars is still a claim, but it is a claim you can argue with.

## Demos are advertisements

A video can be real and still be selected. Cherry-picked prompts, hidden retries, and off-camera tools are ordinary. A live demo that works once is evidence that a path exists, not that the path is reliable. Treat demos the way you treat a car commercial: interesting, not a fuel-economy test.

## Anecdotes are not rates

“It wrote my brief” and “it invented a case” can both be true. They do not tell you how often either happens on *your* workload. For everyday harms that do not need a leaderboard, see [risks without hype](https://aiforthedummies.com/articles/risks-without-hype/). For the difference between training-time learning and a single answer, see [training and inference](https://aiforthedummies.com/articles/training-and-inference/).

## What to do instead of collecting scores

If you must choose a tool, run a small, boring test on work you already know: a handful of real emails, a real spreadsheet, a real policy question. Write down what “good” means before you look. Keep the cases. That notebook will outlast this month’s chart. This site will not publish a fake one on your behalf.

Back: [article index](https://aiforthedummies.com/articles/) · [start of the path](https://aiforthedummies.com/articles/what-ai-is/).