The Year in AI Papers
The year/Independent research

Paper 2605.13841

EVA-Bench: A New End-to-end Framework for Evaluating Voice Agents

Tara Bogavelli, Gabrielle Gauthier Melançon, Katrina Stankiewicz, Oluwanifemi Bamgbose, Fanny Riols, Hoang H. Nguyen, Raghav Mehndiratta, Lindsay Devon Brin, Joseph Marinier, Hari Subramani, Anil Madamala, Sridhar Krishna Nemala, et al.

Published
May 2026
Research lab
Independent
Citations
3
GitHub
192 stars

01 In brief

Summary

EVA-Bench is an end-to-end evaluation framework for voice agents that addresses two challenges: generating realistic simulated conversations and measuring quality across voice-specific failure modes.

It uses bot-to-bot audio conversations with automatic simulation validation, and introduces two composite metrics: EVA-A (Accuracy) for task completion, faithfulness, and speech fidelity, and EVA-X (Experience) for conversation progression, conciseness, and turn-taking.

The framework includes 213 scenarios across three enterprise domains, a perturbation suite for accent and noise robustness, and pass@1, pass@k, and pass^k measurements.

Across 12 systems, no system exceeds 0.5 on both EVA-A and EVA-X pass@1, peak and reliable performance diverge substantially (median pass@k–pass^k gap of 0.44 on EVA-A), and perturbations expose robustness gaps with mean deltas up to 0.314.

The framework is released open-source.

02 From the paper

Abstract

Voice agents, artificial intelligence systems that conduct spoken conversations to complete tasks, are increasingly deployed across enterprise applications. However, no existing benchmark jointly addresses two core evaluation challenges: generating realistic simulated conversations, and measuring quality across the full scope of voice-specific failure modes. We present EVA-Bench, an end-to-end evaluation framework that addresses both. On the simulation side, EVA-Bench orchestrates bot-to-bot audio conversations over dynamic multi-turn dialogues, with automatic simulation validation that detects user simulator error and appropriately regenerates conversations before scoring. On the measurement side, EVA-Bench introduces two composite metrics: EVA-A (Accuracy), capturing task completion, faithfulness, and audio-level speech fidelity; and EVA-X (Experience), capturing conversation progression, spoken conciseness, and turn-taking timing. Both metrics apply to all major agent architectures, enabling direct cross-architecture comparison. EVA-Bench includes 213 scenarios across three enterprise domains, a controlled perturbation suite for accent and noise robustness, and pass@1, pass@k, pass^k measurements that distinguish peak from reliable capability. Across 12 systems spanning all three architectures, we find: (1) no system simultaneously exceeds 0.5 on both EVA-A pass@1 and EVA-X pass@1; (2) peak and reliable performance diverge substantially (median pass@k--pass^k gap of 0.44 on EVA-A); and (3) accent and noise perturbations expose substantial robustness gaps, with effects varying across architectures, systems, and metrics (mean $Δ$ up to 0.314). We release the full framework, evaluation suite, and benchmark data under an open-source license.