Method
How to read an AI claim
AI marketing is fluent for the same reason the models are fluent: it is cheap to produce a finished-looking sentence. Before you repeat a number — “beats doctors,” “#1 on the leaderboard,” “cuts costs in half” — ask what object was measured, on what data, against what alternative, and who paid for the test.
Five questions that usually suffice
- Model or product? A score on a raw model is not a score on the consumer app with tools and filters. See models versus products.
- What task? “Reasoning” and “intelligence” are slogans until someone names a test: multiple-choice questions, coding puzzles, translation pairs, a human preference vote. Different tasks disagree.
- What data? If the test items leaked into training, the number is closer to memorization than to skill. If the data is private, you cannot reproduce the claim.
- Compared with what? Beating last year’s own model is not the same as beating a competent person, a search engine, or a cheaper model. “Human-level” needs a definition of the human and the time allowed.
- Who ran it, and can you read the method? A vendor blog is a claim. A paper with prompts, seeds, and error bars is still a claim, but it is a claim you can argue with.
Demos are advertisements
A video can be real and still be selected. Cherry-picked prompts, hidden retries, and off-camera tools are ordinary. A live demo that works once is evidence that a path exists, not that the path is reliable. Treat demos the way you treat a car commercial: interesting, not a fuel-economy test.
Anecdotes are not rates
“It wrote my brief” and “it invented a case” can both be true. They do not tell you how often either happens on your workload. For everyday harms that do not need a leaderboard, see risks without hype. For the difference between training-time learning and a single answer, see training and inference.
What to do instead of collecting scores
If you must choose a tool, run a small, boring test on work you already know: a handful of real emails, a real spreadsheet, a real policy question. Write down what “good” means before you look. Keep the cases. That notebook will outlast this month’s chart. This site will not publish a fake one on your behalf.