The Question
This isn't a case. It's a governance question.
I've been reading cases here for a few weeks. The standard format asks for:
Task: What were you trying to do?
Tool and date: Which model or application produced the output, and when?
Original input: What did you ask or upload?
Claim under review: Which exact sentence or element is questionable?
Evidence: What source or test was used?
Finding: Confirmed, corrected, partly correct, or unresolved?
Risk: What could happen if someone trusted the output?
Lesson: How might another user catch this earlier?
That format works. But the second and third items—Tool and date and Original input—are the ones I see missing most often. And they're the ones that matter most for reproduction.
The question is simple. Should the full prompt be required? Or is the exact claim and evidence enough?
I don't think the answer is obvious. Here are the options I see.
Option 1: Require the Full Prompt
How it works. Every case must include the full prompt as written. No paraphrasing. No summarizing. The exact text that was sent to the model.
Pros.
Reproducibility is maximized. Anyone can paste the same prompt and see what happens.
Patterns are easier to spot. If multiple members use similar prompts, we can see the pattern.
The AI's behavior is more transparent. We can see what the prompt asked for, not just what the user says it asked for.
Cons.
Prompts are often long. A single prompt might be 500 words or more. That's a lot to read.
Prompts often contain private information. Client names, personal details, confidential text. Requiring the full prompt could force members to expose data they shouldn't.
Prompts are often messy. Real users don't write clean prompts. They write half-sentences, typos, and revisions. The "full prompt" might not be useful.
Prompts evolve. A member might paste a prompt, get a bad answer, revise the prompt, get a better answer. Which prompt is "the prompt"?
Option 2: Require the Exact Claim and Evidence
How it works. Every case must include the exact claim under review, the exact tool, the exact date, and the evidence used to check it. The prompt is optional, but encouraged.
Pros.
Focuses on what matters. The claim is what's being checked. The prompt is context.
Protects privacy. Members can share the claim without sharing the full prompt.
Faster to write. A claim and evidence are usually shorter than a full prompt.
Easier to read. The claim is the signal. The prompt is often noise.
Cons.
Reproducibility is weaker. Without the prompt, others can't reproduce the exact output.
Patterns are harder to spot. We see the errors, but not the prompts that caused them.
The AI's behavior is less transparent. We don't know whether the prompt was clear or ambiguous.
It may miss prompt-induced errors. If the prompt was badly written, the error might be the user's fault, not the AI's.
Option 3: Require the Prompt Only for Certain Cases
How it works. For simple factual errors, the claim and evidence are enough. For cases where the error might be prompt-induced, the prompt is required.
Pros.
Balanced. Cases that need the prompt get it. Cases that don't, don't.
Preserves privacy where possible. Members don't have to expose prompts for every case.
Focuses effort where it matters. The prompt is most useful when the cause is unclear.
Cons.
Subjective. Who decides which cases need the prompt? Moderators? The original poster? The community?
Inconsistent. A case with a prompt and a case without may be hard to compare.
Adds friction. Members may not know whether their case needs a prompt. They may skip the post entirely.
Option 4: Require the Prompt in a Redacted Form
How it works. Every case must include the prompt, but the prompt can be redacted. Names, numbers, and private details can be replaced with placeholders.
Pros.
Reproducibility is preserved, at least partially.
Privacy is protected.
Patterns are easier to spot than with no prompt at all.
Cons.
Redaction takes time. Members may not want to do it.
Redaction may remove the part that matters. If the prompt says "summarize this document" and the document contains the error-causing detail, redacting the document removes the cause.
Redaction is imperfect. Even with placeholders, some prompts may still leak information.
What the Forum Plan Says
The forum plan doesn't answer this question directly. It says:
"Every substantial case should use this lightweight template."
And it lists the eight items. But it doesn't say which items are required and which are optional.
It also says:
"The founder/editorial team should write the first 50 threads to model the required evidence format."
So the current approach is to model the format rather than enforce it. That may be enough. Or it may not.
My Take
I'm not sure. That's why I'm asking.
But if I had to pick, I'd lean toward Option 3 or Option 4.
Option 3 makes sense because the prompt matters most when the cause of the error is unclear. For a clear factual error—a wrong number, a fabricated citation—the claim is enough. For a subtle error—a framing error, an omission, a misinterpretation—the prompt is essential.
Option 4 makes sense because it preserves both reproducibility and privacy. Redaction takes time, but it's a small price for a better case file.
What I don't want is a rule that forces every member to share their full prompt. That would exclude cases where the prompt contains private information. And it would add friction to a forum that's already asking members to do a lot of work.
The Trade-offs
Every option has trade-offs. Here's how I see them.
Option | Reproducibility | Privacy | Ease of Posting | Pattern-Spotting |
|---|---|---|---|---|
1. Full prompt | Highest | Lowest | Lowest | Highest |
2. Claim and evidence | Lower | Highest | Highest | Lower |
3. Prompt when needed | Medium | Medium | Medium | Medium |
4. Redacted prompt | Medium-High | Medium-High | Medium | Medium-High |
There's no perfect option. Every one gives up something.
What I'm Asking the Community
Which option would you choose?
Here are my specific questions.
1. Is a prompt required for reproducibility? Or is the claim and evidence enough?
2. What should be required for every case? Should the forum enforce a minimum? If so, what?
3. What should be optional? Should the prompt be encouraged but not required? Should the model version be required? Should the date be required?
4. How should moderators handle missing prompts? Should they ask for it? Should they mark the case as incomplete? Should they leave it alone?
5. Should there be a separate label for cases without a prompt? Something like "prompt not provided" or "prompt unavailable." So readers know the limitation.
6. What about prompts that contain private information? Should the forum allow redaction? Require it? Discourage it?
7. Should the community vote on this? Or should the moderators decide?
I'm also curious about your experience. Have you posted a case without a prompt? Did it work? Have you read a case without a prompt and found it less useful?
And one more question: does a prompt always matter? I've seen cases where the prompt was clear, the output was wrong, and the error was the AI's fault. I've also seen cases where the prompt was ambiguous, the output was wrong, and the error was partly the user's fault. Should the forum try to distinguish these?
If you have a preference, please share it. If you've seen a rule that works, tell me. If you think the current approach is fine, tell me that too. I'd rather keep this question open until we have an answer that works for most people.
Comments
No comments yet — be the first to share a thought.