AI Guides › Judgement & Guardrails
Building Your Own Error Log Instead Of Trusting From Memory
By Nigel Guy · 2 min read
Ask most people how often their AI tools get things wrong and you'll get a
vague impression instead of an answer — "not often," "mostly for dates," "it
messed up on me once with a citation." Memory is a bad instrument for this.
It's shaped by whichever error was most recent or most embarrassing, not by
what actually happens most often, and a vague impression can't tell you
where to focus your checking.
The rule: your own error patterns are only knowable if you write them
down as they happen — a real log beats a strong impression every time,
because impressions are exactly the kind of thing that get quietly
rewritten by memory.
The mechanism: the error log
Keep it as simple as a running table, in whatever tool you already use for
notes. Five columns are enough:
| Date |
What it got wrong |
How you caught it |
Stakes if you'd missed it |
Pattern |
| — |
The specific error — a wrong figure, an invented source, a skipped step |
The check that actually caught it |
What would have happened if it had gone through unchecked |
Anything this has in common with a previous entry |
- Log it the moment you catch it, not from memory later. Details fade
fast, and the "how you caught it" column is the most useful one — it's
your own working method, documented.
- Be specific about the error type, not just "it was wrong." A wrong
date and a fabricated quote are different failure modes and probably
need different checks.
- Note the stakes honestly. Some entries are trivial; noting that is as
useful as noting the serious ones, because it stops you over-correcting
into checking everything at the same intensity.
- Review the log periodically, not just add to it. The value is in the
pattern column filling in over time — certain topics, certain tools, or
certain kinds of request turning out to be where your own risk actually
concentrates.
- Let the log change your workflow, not just sit there. If three
entries in a row are the same failure mode, that's a standing check
worth building into how you work, not another log entry.
What to skip
Skip logging every minor slip in casual, low-stakes use — a log that's
mostly noise stops getting reviewed. And skip trying to reconstruct a
comprehensive log retroactively from memory; that defeats the entire point.
Start it from today and let it build.
Guardrails
- A short log kept honestly is worth more than a long one kept
inconsistently. The habit of logging matters more than the volume.
- This is a personal instrument, not a scientific study — don't generalise
your own log's patterns to "AI is unreliable at X" as a broad claim
without a much larger, more careful sample than a personal log provides.
- The log only helps if you actually change behaviour based on what it
shows. A pattern you notice and then ignore is just an interesting fact,
not a guardrail.
All 751 AI guides · JulieMango plans from £17/mo