Why a Public Correction Can Build More Trust Than the Original Study

0
24

By 3 August, AI search manager Khadija Zaman had rewritten a study she led twelve times in five days. It had gone live on 29 July, and by any normal measure it was done.

The study looked at how often five AI engines (ChatGPT, Gemini, Perplexity, Google AI Overviews and Google AI Mode) cite the same sources for the same question, across more than 13 million citations.

AI search visibility research is only useful if it holds up, so she re-ran it against a null model. Two findings didn’t survive, and the full record now sits in the AI citation study re-test published by author Khadija Zaman, next to the numbers it replaced.

What the stronger test showed

A null model shows how much agreement you’d get between engines by pure chance. Real agreement minus chance agreement is what a finding is actually worth.

Finding one: brands versus pages

The first claim said brands travel across AI engines 4.5 times better than individual pages. Against chance, the order reversed: page-level agreement was 42 times chance, brand-level 17 times.

Brands only looked stronger because an answer has a handful of plausible brands to pick from and an enormous number of plausible pages.

Finding two: Google’s own surfaces

The second claim said Google’s two AI surfaces agree 3.4 times more than engines from different companies. Against chance, that premium shrank to 1.06x.

A third problem nobody had flagged

The post said every question was collected in the United States. The data said 84.06% United States, across 27 markets.

The decision was a leadership call

Nobody had challenged those findings. The piece was performing, and quietly leaving the numbers alone would have cost nothing that week.

Zaman kept the page up and put the corrections on it, beside the original figures, with a dated changelog. The headline finding, that 79.61% of cited websites appeared on only one engine, held.

It wasn’t free

A corrected study is a less tidy story than an uncorrected one. Some readers only see the changelog and decide the whole thing was wrong.

That’s a real cost. The only answer to it is a changelog written clearly enough to read on its own.

Why corrections matter more in AI search

For a long time, credibility online was partly a function of confidence. The page that stated its claim most boldly, with the most backlinks, tended to win.

AI engines change that incentive. When ChatGPT or Perplexity builds an answer, it chooses which sources to quote, and those choices barely overlap across engines.

Outdated claims aren’t neutral

If a model repeats a figure an author stopped standing behind, the reader gets the old mistake, delivered with confidence and the author’s name attached.

Google’s own guidance on helpful, reliable, people-first content asks whether content is accurate and trustworthy. A dated changelog is a visible answer to that question.

Consistency is part of the signal

The same logic applies to every place a name appears. Keeping reviews, mentions and brand details consistent across channels gives both readers and machines one version of you to trust.

Where to draw the line

Not every fix deserves a public changelog. A typo, a broken link or a date format can simply be fixed.

A useful threshold: could someone have made a decision based on the old version? If yes, the change goes on the record.

This also isn’t an argument for publishing half-finished work and patching it later. The test that was skipped should have run before launch.

The lesson was mostly about sequence: build the null model first, then write the headline. Any thought leader publishing data can apply the same rule.