AI Guides › Trend Watch
Reading Your Own AI Usage Data Like A Researcher Would
By Nigel Guy · 2 min read
Most people have a strong sense of how much AI helps them, and almost none
of them have checked. The impression comes from memorable moments — the
time it saved an afternoon, the time it confidently got something wrong —
and those moments are exactly the ones least representative of normal use.
Meanwhile, the actual record is sitting there in your chat history, your
billing page, and your own calendar, unread.
The rule: treat your own AI use as a small study — define a question,
collect a sample, and look at what the record shows rather than what you
remember — before deciding whether a tool is earning its place.
The Personal Usage Study
- Ask one specific question. Not "is AI useful?" but "which of my
tasks does this tool actually speed up?" or "how often do I have to
redo its output?" One question keeps the study honest.
- Choose a window. Two to four weeks of normal work is enough. Skip
unusual periods like holidays or launch weeks.
- Pull a sample, not everything. Take every fifth conversation, or
all conversations from three chosen days. Reading your whole history
invites cherry-picking.
- Code each item. For each conversation, note a few plain fields:
| Field |
Options |
| Task type |
Writing, research, code, admin, thinking-aloud, other |
| Outcome |
Used as-is, used with edits, abandoned, redone manually |
| Time effect |
Clearly saved, roughly neutral, cost time |
| Verification needed |
None, light, heavy |
- Count before you interpret. Tally the columns. Then look for
patterns: which task types cluster in "used as-is," which in
"abandoned."
- Write a two-line finding. For example: "Most useful for first
drafts of routine emails; research tasks mostly needed heavy checking."
That's something you can act on.
What researchers watch for that you should too
- Recall bias. You remember the dramatic wins and failures. The sample
corrects for that.
- Survivorship. Conversations you abandoned early may not feel like
data, but they are. Include them.
- Confounding. If you used the tool more during a quiet week, time
saved might partly be time you had anyway. Note the context.
- Small numbers. A few weeks of personal data suggests; it doesn't
prove. Hold the finding loosely.
What to skip
- Skip the tool's own "time saved" figures if it offers them. They're
estimates built on assumptions you can't see, and they're rarely
designed to be modest.
- Skip tracking everything forever. A short, focused study once or
twice a year beats a spreadsheet you abandon after four days.
- Skip comparing your numbers to other people's. Their work, tasks,
and standards are different; the useful comparison is you against you.
Guardrails
- Your chat history may contain sensitive material. If you export it for
analysis, store it carefully and delete the export when you're done.
- Coding outcomes is a judgement call. Be consistent rather than precise,
and don't pretend the result is more exact than it is.
- A finding that a tool isn't helping with one task type isn't a verdict
on the tool overall — it's a reason to use it differently or not for
that task.
All 751 AI guides · JulieMango plans from £17/mo