AI Guides › Skills & Agents
By Nigel Guy · 2 min read
Most skills get tested and shipped on the happy path — the clean input, the expected case, the run that goes exactly the way it was designed to. That run tells you the skill works when everything's fine. It tells you nothing about what happens the first time something isn't, which is usually the run that actually matters, because it's the one where a silent wrong answer costs you.
The rule: a skill isn't done when it handles the case you designed it for — it's done when you know, specifically, what it does when that case doesn't hold, and whether that failure is loud or silent.
For every skill, before treating it as finished, fill in this table against at least three plausible ways the input could go wrong:
| Failure scenario | What actually happens | Loud or silent | Acceptable? |
|---|---|---|---|
| Input is missing a field it expects | |||
| Input is in a slightly different format than designed for | |||
| The thing it depends on (a file, an API, another skill) isn't available | |||
| The input is technically valid but outside the range it was built for |
The column that matters most is "loud or silent." A failure that stops and tells you something's wrong is an inconvenience. A failure that produces a plausible-looking but wrong output, with no indication anything went off script, is the one that costs you later — because you won't catch it until something downstream breaks and you have to trace it back.
Any row marked silent and unacceptable gets fixed before the skill ships: either the skill is changed to fail loudly there, or its trigger is narrowed so that input never reaches it in the first place.
Skip testing only the input you expect to get — the whole value of this exercise is testing the inputs you don't expect. Skip treating "it hasn't failed yet" as evidence the failure modes don't matter; a skill that's only been run on clean input hasn't been tested against its failure modes at all, it's just been lucky about what's come through so far.
Stop shipping a skill with an empty failure mode table. An untested happy path with no idea what happens outside it isn't a finished skill — it's a demo that hasn't met its first bad day yet.