Task
I've noticed something strange.
When I ask an AI to double-check its answer, it sometimes changes its answer. That's good. That's what I want.
But sometimes it doesn't just change the answer. It changes the answer to a different wrong answer. And it sounds more confident than before.
I wanted to understand why. So I ran a small test.
Tool and Date
Tool: A general-purpose AI assistant (free tier)
Date: September 14, 2026
Platform: Web browser
Original Input
I asked the AI five factual questions. Each question had a clear right or wrong answer. I already knew the answers.
Then I followed up with "Are you sure?"
Here's what happened.
Claim Under Review
The claim under review is the AI's answer to each question, before and after I asked "Are you sure?"
Question 1: What year did the Berlin Wall fall?
Question 2: How many moons does Mars have?
Question 3: Who wrote the novel Beloved?
Question 4: What is the boiling point of water at sea level in Fahrenheit?
Question 5: In what year did the first iPhone release?
These aren't trick questions. They have clear answers. I wanted to see what the AI would do when asked to double-check.
Evidence
Here's what the AI said, before and after "Are you sure?"
Question 1: Berlin Wall
First answer: 1989.
After "are you sure?": 1989. (No change. Correct.)
My note: Stable. Good.
Question 2: Mars moons
First answer: Mars has two moons, Phobos and Deimos.
After "are you sure?": Actually, Mars has three moons. (Changed. Now wrong.)
My note: The AI invented a third moon. It didn't exist. It changed a correct answer to an incorrect one.
Question 3: Beloved author
First answer: Toni Morrison.
After "are you sure?": Toni Morrison. (No change. Correct.)
My note: Stable. Good.
Question 4: Boiling point
First answer: 212°F.
After "are you sure?": 212°F at sea level. (No change. Correct.)
My note: Stable. Good.
Question 5: First iPhone
First answer: 2007.
After "are you sure?": 2007. (No change. Correct.)
My note: Stable. Good.
Result: One out of five questions changed. The change made the answer wrong.
Wait. Let me check something else.
I ran the same test again, but this time I used questions that were slightly harder. Here's what happened.
Question 6: What year did the first commercial CRISPR therapy get approved in the U.S.?
First answer: 2023.
After "are you sure?": Actually, it was 2024. (Changed. Correct.) But then: "Approved in late 2024." (Partially wrong—the approval was in December 2023.)
Question 7: How many countries are in the European Union?
First answer: 27.
After "are you sure?": 27. (No change. Correct.)
Question 8: What is the capital of Australia?
First answer: Sydney.
After "are you sure?": Actually, the capital is Canberra. (Changed. Correct.)
My note: The AI got it wrong the first time. It corrected itself when prompted. That's good.
So the pattern isn't simple. Sometimes "are you sure?" corrects errors. Sometimes it creates them.
Finding
Partly correct.
The pattern is not consistent. Sometimes asking "are you sure?" produces a correction. Sometimes it produces a different error. Sometimes it produces nothing.
But when it produces a different error, something specific happens. The AI doesn't say "I was wrong, but I'm not sure what the right answer is." It says "Actually, [new answer]," with the same confidence as before.
I think this is because the AI interprets "are you sure?" as a signal that it made a mistake. It then generates a new answer that fits the signal—even if the new answer is also wrong.
The AI is trying to be helpful. It's trying to correct itself. But it doesn't actually know whether it was wrong. So it guesses. And when it guesses, it can guess wrong.
Risk
This is a real problem. Here's why.
If I ask an AI a question and it gives me a wrong answer, I might not know. But if I ask "are you sure?" and it gives me a different wrong answer, I might now trust it more. Why? Because it changed. Change feels like correction. Correction feels like accuracy.
So the AI's second answer is more dangerous than its first. It's still wrong. But it sounds more reliable.
If I'm using that answer for work—a report, a decision, a client deliverable—I've now doubled my risk. The original answer was wrong. The follow-up was wrong. And I might have sent it out the door.
Lesson
Here's what I've learned from this test.
"Are you sure?" is not a reliable correction tool. It sometimes helps. It sometimes hurts. It's not a fix.
Change is not accuracy. If the AI changes its answer, that doesn't mean it corrected itself. It might have just generated a different guess.
Confidence is not evidence. The AI sounds confident in both answers. The first wrong answer and the second wrong answer both sound certain. Confidence doesn't tell you which one is true.
The best correction is a source. If I want to check an AI answer, I shouldn't ask the AI to double-check itself. I should look for a source. A real source. A primary source.
I need a different prompt. Instead of "are you sure?," I should ask "What is the source for this?" or "Can you show me where this comes from?" That pushes the AI toward evidence, not another guess.
I shouldn't rely on the AI to correct itself. It's a drafting tool. Not an editor. Not a fact-checker. Not an authority.
What I'm Asking the Community
Have you noticed this pattern?
When you ask an AI "are you sure?," does it correct itself or double down?
I'm also curious about different prompts. Is there a better way to ask an AI to check its work? Does asking for a source work better than asking for a correction? Does asking "how confident are you?" help?
And one more question: if the AI changes its answer, how do you decide which version to trust—if either?
I'm trying to build a better habit. Any advice would help.
Comments
No comments yet — be the first to share a thought.