Task
I was editing an AI-written article for a client.
Not a big article. About 1,500 words. A general piece about small business trends. The client had generated the first draft with an AI tool and asked me to review it before publishing.
I didn't just want to proofread it. I wanted to know how accurate it was. So I decided to check every factual claim in the article. All 50 of them.
Here's what I found.
Tool and Date
Tool used to generate the draft: A general-purpose AI assistant (client's choice, paid tier)
Date of generation: Not provided by client. Likely September 2026.
Article length: About 1,500 words
Total factual claims: 50 (defined as any specific number, date, name, study, statistic, or verifiable statement)
Original Input
I don't have the original prompt the client used. But based on the article, it was something like:
"Write a 1,500-word article about small business trends in 2026. Include specific statistics, dates, and examples."
The article included statistics, dates, company names, study citations, and trend predictions.
Claim Under Review
I checked every factual claim in the article. Here's how I categorized them.
What counts as a factual claim:
Any specific number (e.g., "40% of small businesses...")
Any date (e.g., "in 2023...")
Any named entity (e.g., "according to a Harvard study...")
Any verifiable statement (e.g., "the average cost of...")
Any prediction presented as fact (e.g., "by 2027, most small businesses will...")
I did not count opinions, general statements, or subjective claims.
Evidence
I checked each claim using four methods.
Method 1: Direct source lookup. If the claim cited a study, report, or organization, I searched for the original source.
Method 2: Reverse search. If the claim included a statistic, I searched for the exact number and phrasing.
Method 3: Date verification. If the claim included a date, I checked whether the event actually happened on that date.
Method 4: Cross-reference. If the claim was about a trend or prediction, I checked whether other credible sources said the same thing.
I spent about 4 hours on this audit. Not because it was hard, but because 50 claims take time.
Findings
Here's the breakdown.
Total claims checked: 50
Accurate: 36 (72%)
Inaccurate: 8 (16%)
Unsupported: 6 (12%)
Total error rate: 14 out of 50 (28%)
Here's what the errors looked like.
Inaccurate claims (8):
Three statistics were wrong (numbers didn't match the original source)
Two dates were wrong (events happened in different years)
Two company names were wrong (the companies exist but didn't do what the article said)
One study was attributed to the wrong organization
Unsupported claims (6):
Four claims had no source at all
Two claims cited sources that didn't exist
What surprised me most: The article didn't feel wrong. It read smoothly. The errors were buried. If I hadn't checked, I would have published it.
Risk
If the article had been published as written, here's what could have happened.
Credibility damage. Readers who know the space would have noticed the wrong statistics. Some would have commented. Some would have shared. The client's reputation would have taken a hit.
Legal risk. Two of the inaccurate claims involved company names. If those companies had seen the article, they could have asked for a correction. In a worst-case scenario, they could have threatened legal action.
Client trust. The client hired me to review the article. If I had missed 14 errors, they would have questioned my value. If they had found the errors themselves, they might have wondered why they hired me.
SEO risk. Search engines don't directly penalize inaccurate content. But if readers bounce because the content is wrong, rankings drop.
The risk wasn't just embarrassment. It was professional and financial.
Lesson
Here's what I learned from auditing 50 claims.
72% accuracy is not good enough. The article was mostly accurate. But 28% error rate is too high for anything that's going to be published under a client's name.
Errors cluster around specifics. The AI got the general trends right. It got the specific numbers, dates, and names wrong. That's the pattern.
Smooth writing hides errors. The article read well. The errors didn't jump out. They were buried in otherwise accurate paragraphs.
You can't spot-check an AI-written article. I checked all 50 claims. If I'd only checked 10, I would have missed most of the errors. The errors weren't in the most obvious places.
The client didn't know. The client sent me the draft and said "can you clean it up?" They didn't know it had 14 errors. They assumed it was accurate.
A four-hour audit isn't scalable. I can't spend four hours auditing every AI-written article. I need a faster process. That's why I'm asking the community.
What I'm Asking the Community
What error rate would concern you?
If you audited an AI-written article and found 28% of the factual claims were wrong or unsupported, would you publish it? Would you rewrite it? Would you send it back?
I'm also curious about acceptable thresholds. Is 5% acceptable? 10%? 0%? And does it depend on the type of content?
I'd also like to know: what's your fastest process for auditing a long AI-written document? I spent 4 hours on 50 claims. That's about 5 minutes per claim. Can it be faster?
Finally: what would you tell the client? How do you explain that their AI-generated draft has 14 errors without making them feel like they wasted their money?
I'm trying to build a faster, more scalable audit process. Any advice would help.
A Few More Details
I should mention: I didn't include the client's article or name. I've anonymized the details. The error rate is real, but the specific content is not.
I should also mention: I'm not against AI-written content. I use AI for drafting myself. But I've learned that AI drafts need a real audit before they go anywhere with a client's name on them.
If you have a faster method, please share it. If you've done a similar audit, I'd like to compare notes.
Comments
No comments yet — be the first to share a thought.