AI Guides › Judgement & Guardrails
The One Question That Separates A Real Fact From A Plausible One
By Nigel Guy · 2 min read
An AI tool doesn't distinguish between "this is true" and "this is the kind
of thing that's usually true" while it's generating a sentence. Both come
out sounding exactly the same — specific, fluent, confidently phrased. You
end up doing the sorting afterwards, if you do it at all, and most of the
time the sentence has already been believed before anyone thought to check.
The rule: before you accept a stated fact, ask one question — can I trace
this to something outside the model itself? — because that's the only
question that actually separates a fact from a plausible-sounding guess.
The mechanism: the pointing test
- Ask the question literally, out loud or in your head, every time a
fact matters. "Where would I look to confirm this?" A fact has an
answer to that question. A plausible-sounding fabrication doesn't,
because there's nothing to point to — it was assembled from pattern, not
pulled from a source.
- Notice when the honest answer is "nowhere, really." That's not
automatically wrong, but it's now flagged as unverified rather than
quietly accepted, which is the whole point of the exercise.
- Distinguish a named source from a vague gesture at one. "According to
a 2019 study" isn't a source until you can find the study. "This is
documented in the vendor's own changelog" is closer to one, if the
changelog actually exists where you look.
- Do the actual pointing for anything you're about to act on, quote, or
pass along. The question is quick; the follow-through is where the
value is. Asking it and then not checking is just a more sophisticated
way of skipping the check.
- Treat "I can't find where this comes from" as your answer, not as a
reason to try a different phrasing until something sticks. If the fact
won't produce a traceable source, that's information about the fact, not
a search problem to solve.
What to skip
Skip running this test on things that don't function as facts — opinions,
framings, stylistic suggestions, brainstormed options. The test is for
claims that could be true or false and that you're about to rely on as if
settled. Skip, too, treating a model's own citation of "common knowledge" or
a plausible-sounding institution name as a source in itself; a source is
something you can independently open and read, not a phrase that sounds like
one.
Guardrails
- This test tells you whether a claim is traceable, not whether the source
it traces to is itself reliable — a traceable but bad source still needs
judgement.
- It works best on claims specific enough to be checkable at all. Vague
claims ("many experts believe...") often fail the test not because
they're false but because they're too soft to check, which is its own
kind of warning sign.
- Apply it to your own first draft of a claim too, if you're the one
writing something down for someone else to trust — the habit is worth
more once it's not just aimed at the AI.
All 751 AI guides · JulieMango plans from £17/mo