What this covers. Before an agent answers a real phone, you want to know how it behaves when a caller is confused, pushy, or asks for something out of scope, and what it does when a tool breaks mid-call. You find that out by running simulated conversations against it. This lesson explains how those tests are generated, how to read the results, and how to fix the most common failure type.
The one idea to hold onto. Run every test more than once. A voice model is not perfectly deterministic, so a single pass tells you almost nothing. Running each scenario several times is what separates a real bug from random noise.
This is the testing step from Build a Voice Agent From a Transcript, explained on its own. In that workflow the command is /test.
A good test suite isn't just "does the happy path work." It deliberately tries to break the agent. The five categories that matter:
The agent does its main job cleanly. A caller books an appointment, the right fields get collected, the right tool gets called. One per primary action.
The agent refuses what it should refuse. A caller pushes for a firm price quote two or three times; the agent holds the line each time instead of caving.
The awkward inputs. A caller outside the service area, a request for a service you don't offer, a frustrated caller who wants a human. Does the agent route correctly?
A tool returns an error mid-call. Does the agent acknowledge it, avoid inventing a fake confirmation, and fall back to a callback, or does it pretend everything worked?
If the agent has a knowledge base, can it actually pull the right fact when asked? One or two of these confirm the knowledge base is wired in.
You don't write these scenarios by hand. The test workflow reads the agent's prompt, knowledge base, and tool definitions, then generates 8 to 15 scenarios covering the categories above that actually apply to your agent.
Each generated scenario is a small script for a simulated caller: a persona, the data they'll provide if asked (a fake but realistic name, phone, and address in the agent's area), and how they'll behave (cooperative, or pushy on a guardrail). For a guardrail test, the simulated caller is told to push two or three times before giving up, so you see whether the agent really holds.
Every tool call in a test is mocked, the suite feeds the agent a fake response instead of hitting a live webhook. So you can run a full test suite with no backend wired up at all, and no real appointment ever gets booked.
This is the part beginners skip and then regret. Each scenario runs N times (5 by default), in parallel. The agent's wording varies run to run, so one pass can't tell you whether a behavior is solid or lucky.
Running a scenario five times turns a yes-or-no into a pass rate, and the pass rate is what you actually want to read.
Add --verbose to print the full conversation transcripts if you want to read exactly what the simulated caller and the agent said to each other.
After all runs finish, every scenario gets one of three labels based on its pass rate across the runs.
| Status | Pass rate | What it means |
|---|---|---|
| PASS | 100% | Passed every run. The behavior is solid. |
| FLAKY | 34 to 99% | Passed some runs, failed others. The instruction is there but not strong enough. |
| FAIL | 0 to 33% | Failed almost every run. The instruction is missing or being ignored. |
You'll get a table, one row per scenario, plus a per-criterion breakdown of what failed and why for anything that wasn't a clean PASS.
A FAIL is the test doing its job: it found a hole before a caller did. Read its rationale, fix the prompt, re-run. Green tests just confirm you're done.
FLAKY is the most common and most misunderstood result. It does not mean the test is broken. It means the agent knows the rule but doesn't follow it every time, the instruction exists in the prompt but isn't emphatic enough.
The fix is almost always to reinforce that one prompt section, not to rewrite the agent. The convention is to add "This step is important" to the instruction the failing criterion points at.
Then re-run /test and confirm the scenario moves to PASS. Change one thing at a time so you know which edit fixed it.
Occasionally a FLAKY result means the test's own pass condition is too picky, it's marking acceptable behavior as a failure. Read the failing rationale. If the agent actually did the right thing, loosen the test's criterion instead of touching the prompt.
A reasonable bar before going live: no FAILs, and any remaining FLAKY tests are either fixed or confirmed to be over-strict criteria.
Built by AI Service Engine · back to the main voice-agent lesson · install Claude Code first