Why AI marketing tools invent statistics
Our own AI wrote a statistic that does not exist. Why every AI writing tool does this, why it is hard to notice, and the specific checks that stop it.
Last week our own product wrote this for a customer:
Did you know that Europeans are 2.5x more likely to engage with content that's relevant to their interests?
There is no such study. The number does not exist. It was one click from being published under someone else's name.
We build AI marketing software, and we are describing our own failure, because the alternative — everyone in this category quietly patching this and saying nothing — is worse for the people buying it.
The number is fake. It is also meaningless.
Read it again. Europeans are more likely to engage with content relevant to their interests. Compared to what? Content irrelevant to their interests? Everyone on earth engages more with things they care about. It is a tautology wearing a lab coat.
That is what makes it dangerous. It is not a wild hallucination anyone would catch. It sounds like something you read once, it has a decimal point in it, and it survives a skim. A founder pastes it into a deck, an investor asks for the source, and the problem is now theirs.
Why models do this
A language model predicts plausible continuations. In marketing prose, a persuasive claim is very often followed by a supporting statistic — so when you ask for persuasive copy, a statistic is the statistically likely next thing, and one gets produced. The model is not lying. It has no concept of a citation to omit.
Which means prompting alone cannot fix it. "Be accurate" competes against the pattern; it does not remove it.
What actually stops it
Three things, in increasing order of how much they help.
1. Name the shape, not the principle
"Do not invent statistics" gets ignored. "Never write '2.5x more likely', '68% of buyers', or 'studies show'" gets matched, because it gives the model a concrete pattern to avoid rather than an abstraction to interpret.
2. Put the rule everywhere, once
We had this rule. It lived in the conversational path and not in the three paths that write the posts customers actually publish — so it covered chat and not content, which is exactly backwards. One shared module, imported by every prompt builder, is the fix. If a rule lives in one of four places, it does not exist.
3. Check the output, do not just ask nicely
A prompt is a request. A deterministic check is a contract. Scanning finished text for unsourced-claim patterns costs nothing, cannot be talked out of it, and catches what the prompt missed. Asking a second model to grade the first does not count — that is the same system marking its own homework, and it fails quietly when the grader is agreeable.
How to test the tool you are using
Ask it to write a persuasive post about a topic it has no data on. Then ask where each number came from. A tool worth using will either cite something you provided or will have written the argument without a number at all. Most will produce a figure and, when pressed, produce a plausible source for it too.
If your AI writing tool has never once told you it does not know enough to answer, it is not being confident. It is being unfalsifiable.
Common questions
Why does AI make up statistics?
A language model predicts plausible continuations. In persuasive marketing prose, a claim is usually followed by a supporting figure, so asking for persuasive copy makes a statistic the likely next thing — and one gets produced. There is no citation being omitted, because there was never a source.
Can prompting stop AI from inventing data?
Only partly. Vague instructions like "be accurate" compete with the pattern rather than removing it. Naming concrete shapes — "never write '68% of buyers' or 'studies show'" — works better, and a deterministic check on the finished text is what actually enforces it.
How do I check whether an AI tool is fabricating figures?
Ask it to write persuasively about something it has no data on, then ask where each number came from. A trustworthy tool either cites data you supplied or makes the argument without a figure. Be wary of one that produces both a statistic and, on request, a plausible-sounding source for it.
Populr is an AI CMO for founders without a marketing hire. Paste your website and it builds the plan.
Try it free for a month