Dynamic Persona Testing for Voice AI Agents: A 50-Call A/B Study

With Dynamic persona on, the caller's mood moves during the call. Across 25 matched pairs, call length barely changed (15.0 turns off, 14.3 on), but caller actions rose from 34 to 92 and hang-ups from 0 to 14. The extra actions exposed agent failures the frozen caller could not reach.

Voice AI A/B Testing Methodology: 50 Calls Across 25 Matched Pairs

50 calls in 25 matched pairs. Each pair runs the same agent, scenario, persona and dataset row twice, and only the Dynamic persona switch changes. Four blocks run in ABBA order so drift lands on both arms equally.

The set covered 8 personas, 4 scenarios and 3 agent variants: bare English, hardened English and bare Hindi (five Hindi pairs). Both validity checks passed. No call ran in the wrong arm (0 of 50), and no switch-off call produced a mood-gated action (0 of 25).

Dynamic vs. Static Personas: Comparing Caller Actions

Barge-in reads the mood in both arms, which is why it is the control. The zeros in the off column are by design.

How Voice AI Agents Responded to Interruptions and Escalation Requests

How Voice AI Agents Responded to Interruptions and Escalation Requests

OffOn

Stopped (barge-in)

32

34

Talked over (barge-in)

0

2

Ignored a request for a human

0

11

Offered a transfer

0

8

Declined a transfer

0

3

Did not repeat when asked

0

7

Repeated when asked

0

3

Kept talking, came back, checked in

0

4

Unlabelled

0

20

50 calls, 25 matched pairs, run of 21 to 22 Oct 2026.

Data note. 20 agent reactions in the switch-on arm have no label. They are excluded from the counts above. Before any rate is quoted, the 11 ignored requests and the unlabelled reactions need a listen.

Why Voice Agent Task Completion Scores Dropped During Dynamic Persona Testing

The arm in which every handling defect above was found scores 16 points lower. The judge appears to score how satisfied the caller sounded, so a caller who pushed back and left reads as the agent's failure. This is the same inversion seen on 14 October, reproduced here with one variable changed. It is a judge problem to fix before the score is used on stress runs.

A/B Test Predictions vs. Actual Voice Agent Testing Results

Prediction 6 failed for a useful reason. The cheerful control escalated on the ₹5400 read-back: the agent said "Five 1400", and the caller went from calm 3 to frustrated 3 with the reason "wrong amount stated". The feature caught a real defect and tied it to the turn that caused it. That was the result we wanted, but not the prediction we made.

Key Recommendations for Voice AI Testing and Evaluation

1. Keep Dynamic persona on for discovery, where the goal is to find handling defects.
2. Keep it off for deterministic regression suites, where a moving mood makes runs non-comparable.
3. Fix the ignored transfer requests. 11 of 25 went unanswered.
4. Fix the number read-back. It escalated the calmest persona.
5. Validate the Task Completion judge against human labels before using it on stress runs.
6. Label the unlabelled reactions, then listen to the 11 ignored requests.

Additional Findings From the Voice AI A/B Test

• Only 3 of 50 calls fell back to the heuristic judge, so the comparison is almost entirely LLM-judged.
• Hardened prompts did not end calls early. They ran longer.

Study Limitations and Testing Considerations

• One run. Each persona and scenario cell ran once per arm, and caller actions are probabilistic. Treat every figure as a count, not a rate.
• One organisation's production account, one agent, five Hindi pairs.
• Automatic labels were not audited by hand, and 20 reactions have no label.