Task
I was using AI to summarize a vendor's security documentation.
Not for a client. For my own small business. I was deciding whether to switch to a new vendor for handling customer data. The vendor published a 12-page security overview. I asked AI to summarize it so I could make a decision faster.
The summary was accurate. Every sentence matched the source. But it left out one caveat that changed my decision entirely.
Here's what happened.
Tool and Date
Tool: A general-purpose AI assistant (free tier)
Date: September 11, 2026
Platform: Web browser
Document length: 12 pages (about 6,000 words)
Original Input
Here's the exact prompt I used.
"Summarize this vendor security overview. I'm a small business owner deciding whether to use this vendor to handle customer data. Summarize the key points about security, compliance, data storage, and any conditions or exceptions."
I asked specifically for "any conditions or exceptions." The AI still missed one.
Claim Under Review
The AI summary was about 300 words. It covered:
Encryption standards
Data center locations
Compliance certifications
Data retention policy
Access controls
Breach notification process
Every sentence in the summary was factually accurate. I verified each one against the original document.
But the summary omitted one caveat. The original document included this sentence, buried on page 9:
"Data redundancy is provided for all accounts on the Enterprise plan. Accounts on the Standard plan use a single-region storage configuration."
I was planning to use the Standard plan. The summary didn't mention the distinction.
Evidence
I read the original document carefully after reading the summary.
The summary described "data redundancy" as a feature. That was accurate for Enterprise. But for Standard, there was no redundancy. The data was stored in a single region.
For a small business handling customer data, this distinction mattered. If the region went down, my data would be inaccessible. If I'd signed up for Standard based on the summary, I would have misunderstood the risk.
I also found two other caveats the summary missed.
Caveat 2: The compliance certifications applied to the Enterprise plan only. The Standard plan was compliant with a narrower set of standards. The summary said "compliant with SOC 2 and ISO 27001" without the qualifier.
Caveat 3: The breach notification process was described as "within 72 hours" for Enterprise, but "within 5 business days" for Standard. The summary used the Enterprise number.
All three caveats were on pages 7-10 of the 12-page document. All three were buried in paragraphs that the AI summarized as general features.
Finding
Partly correct — factually accurate but materially incomplete.
The summary was not wrong. Every sentence was correct. But it was incomplete in a way that changed my decision.
For a general reader, the summary was fine. For my specific decision, it was misleading. It described the best case (Enterprise) and omitted the conditions that applied to my case (Standard).
This is a specific type of AI failure: accurate but not applicable. The AI summarized the document as if all features were universal. It didn't read for the specific reader's situation.
Risk
The specific risk here: I could have signed up for Standard expecting Enterprise-level protections, then discovered the difference only when something went wrong.
The larger risk is general. If someone uses AI to summarize a contract, a policy, a medical document, or a legal agreement, and the AI omits a caveat that applies to their situation, they might make a decision based on incomplete information.
The summary said everything correct. That's what makes the omission dangerous. A wrong summary is easy to catch. An incomplete summary is easy to trust.
Lesson
Here's what I learned from this case.
"Accurate" is not the same as "complete." A summary can be factually correct and still be misleading. I need to check both.
Look for the specific caveat that applies to my situation. I asked the AI for "conditions and exceptions," but I didn't ask it to look for the caveats that applied specifically to the Standard plan. If I'd asked "what applies to the Standard plan specifically?", I might have gotten a different summary.
The AI summarizes the document, not the decision. The AI treated the document as a whole. It didn't know that I was planning to use Standard. It didn't know that the distinction mattered. I need to include my specific situation in the prompt.
Omissions cluster in the middle. The three caveats were on pages 7-10 of 12. Not at the beginning, not at the end. In the middle, where attention drifts.
Long documents are risky. A 12-page document has more places for caveats to hide. A 2-page document has fewer. The longer the document, the higher the risk of material omission.
Ask for omissions directly. After reading the summary, I now ask: "What conditions or exceptions did you leave out?" The AI often knows what it summarized out. This catches some, not all, of the omissions.
Read the original for high-stakes decisions. For a low-stakes summary, the AI output is fine. For a decision that affects my business, I need to read the original.
What I'm Asking the Community
How should omission be rated?
If the summary was factually accurate but omitted a caveat that changed the decision, is that:
Confirmed error (the AI failed)?
Partly correct (the AI was accurate but incomplete)?
Unresolved (we can't tell without knowing the reader's situation)?
Something else?
I used "partly correct," but I'm not sure that's the right label. The AI didn't get anything wrong. But the summary was misleading for my situation. That feels worse than "partly correct."
I'm also curious about the general pattern. Have you seen AI summaries that were accurate but incomplete? How do you catch the omissions? Do you ask the AI for "conditions and exceptions"? Do you ask specifically for what applies to your situation? Do you read the original for high-stakes decisions?
And one more question: should the forum have a specific error label for "accurate but incomplete"? It feels different from a hallucination and different from a wrong number. Maybe it deserves its own category.
I'd like to know how others rate this kind of failure. I'd rather label it right now than after another material omission slips through.
Comments
No comments yet — be the first to share a thought.