Discovery vs. Regression Testing for Voice AI Agents: Key Differences and Best Practices

A voice agent test suite has two jobs that pull in opposite directions. One finds defects you did not know about. The other proves that the change you just shipped did not break what already worked. They need different callers.

Switch on: discovery. Finds defects. The caller behaves like a person, so it asks for a human, asks for a repeat and sometimes hangs up. Counts move between runs.

Switch off: regression. Shows what else moved. The caller holds one mood, so two runs compare turn by turn and a difference points at the agent.

What Is Discovery Testing for Voice AI Agents?

A discovery run asks one question: what does this agent do when the caller behaves like a person? The output you read is the list of caller actions and the agent's reaction to each. In our last run that was 92 actions across 25 calls, including 11 requests for a human the agent ignored and 7 repeats it failed. Those are tickets to fix, not scores to trend.

Expect two things. Counts move between reruns, because actions are probabilistic, so run a scenario several times before you call a number a rate. And the headline score tends to drop, because the caller is harder to satisfy. That is covered in the fourth post in this series.

What Is Regression Testing for Voice AI Agents?

A regression run asks a different question: is the agent still doing what it did before? For that, every variable except your change has to stay fixed, and the caller is the largest variable on the call.

With the switch off, the caller holds one mood for the whole conversation. Two runs of the same scenario are comparable turn by turn, and a difference points at the agent.

A number read-back is a good example. It has one correct output, so a frozen caller is the right tool there. A caller who hangs up halfway would only get in the way of the check.

In our matched run the frozen arm averaged 15.0 turns against 14.3 for the moving arm. Call length is not what the switch changes. Comparability is.

Discovery vs. Regression Testing: Key Differences

This is the split we use in this series. It is a design choice we have made, not a cited industry standard. Discovery is exploratory: varied, adversarial, meant to find behaviour nobody predicted. Regression is fixed: the same prompts and the same grader, run on every candidate, so two versions can be compared on one number. Keep them as separate suites with separate owners.

The step most teams skip is turning a discovery finding into a regression test. A discovery run finds that the agent ignores a transfer request under pressure. The fix ships. Nobody adds "caller asks for a human at turn 6" to the regression suite, so the next prompt change can reintroduce the defect unnoticed.

Our rule: every defect a Dynamic persona run finds becomes a frozen-persona test. Discovery feeds regression. Regression does not feed discovery.

How to Combine Discovery and Regression Testing in Voice AI QA

1. Discovery pass, switch on. Run each live scenario against several Dynamic personas. Each ignored request, failed repeat and hang-up becomes a ticket with the call attached.
2. Fix during the week. Change the prompt, tool or model, whichever the ticket points at.
3. Before each deploy, switch off. Run the regression suite with frozen callers. Compare turn by turn with the last green run. A difference is a blocker.
4. After the fix lands, write the frozen test. The exact persona, scenario and turn where the defect appeared, with the mood pinned. It runs on every deploy from then on.
5. Repeat. The regression suite grows by one test per defect found. Discovery keeps looking for the next one.

The one thing to resist is running discovery before a deploy and reading the score drop as a regression. It is not a regression. It is a different caller.