Most solo operators ship an AI agent the same way. It worked three times in the builder, so it goes live. Then the client finds the fourth case, and you hear about it in an email that starts with “hey, quick question.”
An AI agent test plan is a written set of inputs, expected outputs, and pass conditions you run against an agent before it touches live work. That is the whole definition. It is a document, not a tool, and it is what separates believing your build works from proving it.
I spent 16 years in enterprise automation before I built anything for my own clients. In that world nothing reaches production without a test script somebody signed. Solo builders dropped the habit along with the paperwork, and it costs them every time.
Key takeaways
- An AI agent test plan is a written set of inputs, expected outputs, and pass conditions run against an agent before it handles live work.
- Testing an agent on data you invented proves nothing. Invented data is always well formed. Client data is not.
- n8n data pinning saves a node’s output and reuses it on later test runs, and it is not available for production executions (n8n, 2026).
- Make scenario run replay reruns a current scenario using trigger data from a past run, on all plans, and consumes credits like a standard run (Make, 2026).
- Microsoft Copilot Studio ships seven built in agent evaluation methods plus a custom one, including general quality, content safety, tool use, and exact match (Microsoft, 2026).
- A pass bar written after the test run is not a pass bar. Set the number before you execute.
Why “it worked when I tried it” is not proof
When you test your own build, you feed it the input you had in mind when you built it. That is not a test. That is a demo.
Real inbound data is messy in specific, boring ways. A name field with a trailing space. A photo of a receipt instead of a PDF. A form submitted twice because somebody double clicked. An agent that reasons its way through a clean record will invent its way through a broken one, and the output still looks like a real answer.
That is the part people miss. A traditional script crashes on bad input, and a crash is a signal. An agent does not crash. It produces something plausible and moves on, and nobody sees the failure until a human downstream notices the numbers are wrong.
The RANGE Test: a five step AI agent test plan
In the Army, nobody hands you a weapon and tells you to go use it. You qualify on a range first, against a known target, scored against a number somebody wrote down before you fired. Then you carry it. Run your agents the same way.
R: Replay real data, not sample data
Pull real records from the client’s system, strip anything sensitive, and use them as test inputs. Twenty real records beat two hundred you made up.
Every major platform supports this now. In n8n, data pinning saves a node’s output and reuses the saved data on later test runs instead of fetching fresh data, so every run hits the same input and you can tell whether a change helped. Pinning only works on nodes with a single main output, it does not accept binary data, and it is not available for production executions (n8n, 2026). In Make, scenario run replay reruns the current version of a scenario using trigger data from a previous run, which is how you test against a payload that already caused a problem. It is available on all plans, and the replay consumes credits exactly like a standard run (Make, 2026). In Zapier, you can select a previous Zap run as the trigger test record instead of the generic sample the app hands you (Zapier, 2026).
Action step: before you build a single node, collect ten to twenty real records and save them where every future test run can reach them.
A: Attack the edge cases
Now break it on purpose. The edge case list is short, and it is the same list for almost every build:
- Empty input. A record with the key field blank.
- Malformed input. Wrong file type, broken date format, text where a number belongs.
- Oversized input. A document ten times longer than anything you tested, which is how you find your context limit.
- The same input twice. Retries are normal, and an agent that cannot tell a retry from a new job will send duplicate actions to a real customer.
- An input outside scope. Something the agent has no business handling, to confirm it stops instead of guessing.
That last one matters most. An agent with no stopping condition will attempt every job you hand it. Your AI agent escalation rules are what get tested here, and if you have not written any, the edge case test is where you find that out.
Action step: write the five cases above into your test document with the expected output for each, then run them.
N: Neutralize the side effects
A test that sends a real email is not a test, it is an incident. Point every write action somewhere harmless first.
Swap client email addresses for your own. Point database writes at a copy. Replace the payment step with a logging step. The rule is simple: during testing, nothing the agent does should be visible to anyone outside your own screen.
This step gets skipped under deadline pressure, and it is the only RANGE step that can damage a client relationship by itself.
Action step: list every outbound action in the workflow and the safe destination each one points at during testing.
G: Gauge against a written pass bar
Decide what passing means before you run anything. A pass bar written after you see the results is not a pass bar, it is a rationalization.
Three numbers are enough. Accuracy: how many of the twenty records produced the correct output. Escalation: how many out of scope cases correctly stopped and handed off. Cost: what one run actually costs you, because an agent that is accurate and loses money is still a failed build. If you have never measured that one, start with AI agent cost per run before you quote anybody.
If your stack supports scored evaluation, use it. Microsoft Copilot Studio ships seven built in evaluation methods plus a custom one. They include general quality, which uses a large language model to score relevance, groundedness, completeness, and abstention, along with content safety, tool use, keyword match, and exact match (Microsoft, 2026). Start with tool use. It checks whether the agent actually called the tool you built instead of answering from memory.
Action step: write your three thresholds at the top of the test document, before the first run.
E: Evidence file
Save the results. Date, record count, pass rate per category, the failures, and what you changed. One page is fine.
This file does three jobs. It is what you hand a client at launch, which turns “trust me” into a document. It is your baseline when the agent behaves differently in six weeks, because you can rerun the same records and see what moved. And it is what lets you stop checking every output and audit by exception instead.
Action step: name the file with the date and the agent, store it with the build, and rerun it after every meaningful change.
Example scenario: an invoice intake agent
Example scenario, written to show the shape of the work, not a client result. A bookkeeping client wants an agent that reads emailed invoices, pulls vendor, amount, and due date, and writes a row to a sheet.
R. You export 20 real invoice emails from the last quarter, redact account numbers, and pin them as the trigger input.
A. You add five deliberate problems: an email with no attachment, a scanned photo instead of a PDF, a 40 page contract with one invoice buried in it, the same invoice forwarded twice, and a personal receipt that is not a vendor invoice.
N. The sheet ID points at your own test sheet. The notification email points at your inbox, not the bookkeeper’s.
G. Written before the run: 18 of 20 extracted correctly, all 5 edge cases handled or escalated, cost per invoice under a figure that still leaves margin on the retainer.
E. One page of results, including the two that failed and why.
Here is what a test like this usually surfaces. The scanned photo extracts a wrong total and writes it confidently. The 40 page contract blows the context window and returns a partial answer with no error. The duplicate forward writes a second row. None of the three show up in a clean demo. All three show up in week one of live use.
Frequently asked questions
How long should an AI agent test plan take?
For a single workflow agent, plan on two to three hours the first time and under an hour on every rerun. Most of the first pass is collecting and redacting real records, and that work is reusable for the life of the build.
Do I need a test plan for an agent I built for myself?
Yes, and the reason is upkeep, not embarrassment. Your own agents drift the same way client agents do when a model updates or a vendor changes an endpoint. A saved set of test records is the cheapest way to find out something broke before it costs you a month of bad output, which is the automation debt nobody prices in at build time.
What is the difference between testing an AI agent and testing a normal automation?
A normal automation is deterministic, so one pass per path is usually enough. An AI agent can return a different answer to the same input, so you test the same record more than once and check the reasoning path, not just the final value. That is why a method like tool use exists: it confirms the agent took the route you designed rather than landing on an acceptable answer by another road.
Can I run an AI agent dry run against live data safely?
You can read from live data safely. You should not write to it. An AI agent dry run means live inputs with every outbound action redirected somewhere harmless, which is the N step in the RANGE Test. Read only plus neutralized writes gives you realistic input with no blast radius.
What if the agent passes my test plan and still fails in production?
Then your record set was missing a case, and that case becomes a permanent part of it. A test plan is a ratchet, not a promise. Every production failure gets added so it cannot surprise you twice.
Recap
An AI agent test plan is a written set of inputs, expected outputs, and pass conditions you run before an agent touches live work. The RANGE Test is the five steps: replay real data, attack the edge cases on purpose, neutralize every side effect, gauge against a pass bar you wrote in advance, and keep an evidence file you can hand to a client and rerun later.
A script breaks loudly. An agent is wrong quietly. A test plan is how you make the quiet failure loud enough to catch on your screen instead of the client’s.
Pick one agent you already have running. Write the five edge cases. Run them today. You will find something.
References
Make. (2026). Scenario run replay. Make Help Center. https://help.make.com/scenario-run-replay
Microsoft. (2026). Choose evaluation methods. Microsoft Copilot Studio documentation. https://learn.microsoft.com/en-us/microsoft-copilot-studio/analytics-agent-evaluation-overview
n8n. (2026). Pin and mock data. n8n Docs. https://docs.n8n.io/build/work-with-data/pin-and-mock-data
Zapier. (2026). Use previous Zap runs as trigger test records. Zapier Help Center. https://help.zapier.com/hc/en-us/articles/35632951078029-Use-previous-Zap-runs-as-trigger-test-records
Leave a Reply