Beyond WER and MOS: What Voice AI Benchmarks Miss in the Real World

WER and MOS miss what breaks real voice agents: meaning, accents, code-switching, tone, latency and task success. Here’s what to measure instead.

Stochastic / Blog

What Voice AI Benchmarks Miss in the Real World

Beyond WER and MOS


Picture a voice agent built from best-in-class parts. Its speech recognizer sits near the top of the Open ASR Leaderboard. Its voice ranks high in a blind TTS arena. Its language model aces reasoning evals. Then a real customer calls from her car on speakerphone, with the radio on and her kids talking in the back seat. She reads out her account number and email address. The agent mishears the email, treats her son's voice as a new request, and cuts her off mid-sentence. She ends up repeating herself three times before asking for a human. Every component passed its benchmark. The call failed anyway.

None of these failures shows up in a word error rate (WER) or a mean opinion score (MOS). Most public benchmarks use clean audio from a single speaker, score each sentence on its own, and count every word as equally important. Real calls are noisy, crowded, full of numbers and jargon, and judged by one question: did the caller get what they needed?

This post walks through six gaps between what voice benchmarks measure and what callers experience: meaning, the words that matter most, the caller's surroundings, tone, timing and task completion. For each one, we look at what current benchmarks capture, what recent research has found, and what to measure instead.

The evaluation map

A voice agent is built in one of two ways. A pipeline chains three models: speech-to-text, a language model, and text-to-speech. A speech-to-speech agent uses one model that listens and replies directly. Either way, each part can be scored on its own, but the caller only ever experiences the whole call.


Scores for each part tell you which part to fix. Only testing whole calls tells you whether callers are actually getting a good experience, because small errors in each part add up, and some problems (like talking over the caller) only appear when the parts run together.

WER scores words, users care about meaning

Word error rate counts substitutions (S), deletions (D) and insertions (I) against the number of words in the reference transcript (N):


Because it treats every word the same, WER fails in two directions. First, the score depends heavily on how text is normalized before comparison. Standard normalizers, such as Whisper's English normalizer, convert numbers, currency, casing and punctuation to a common form, so "$25" and "twenty-five dollars" match. The Open ASR Leaderboard applies standardized normalization to every model for exactly this reason[1]. But normalizers differ in what else they standardize, such as compound words ("log in" vs. "login"), contractions, filler words and spelling variants. Two WER figures computed with different normalizers aren't comparable, so a vendor's headline number may not match what you measure on your own data. Normalization can also hide errors that matter downstream, because your language model reads the raw transcript, not the normalized one.

Second, WER barely notices errors that flip the meaning: "can" vs. "can't" is one substitution in a long sentence. In clinical dialogue, researchers found that WER and existing semantic metrics correlate poorly with the clinical impact experts assign to an error, since a single dropped negation can invert a diagnosis[2].

How researchers are measuring meaning

There is no single replacement for WER. Instead, researchers have developed several families of methods, each covering a different blind spot:

Approach

What it checks

Where it falls short

Word-level error rates (WER, CER), with text normalization

Exact words, after standardizing numbers, case and punctuation

Every word counts the same; results depend heavily on the normalizer

Targeted entity and keyword metrics, such as missed entity rate or medical entity recall [3]

Whether the high-stakes words (names, numbers, emails, terms) came through

Needs labeled entities for your domain

Multiple-reference scoring, such as the lattice of valid alternatives used in the Voice of India benchmark [4]

Accepts every valid spelling or segmentation instead of one "correct" transcript

Still word-level; building the alternatives takes effort

Embedding similarity, such as Semantic Distance, BERTScore and SeMaScore [5]

How close the meaning is, not the surface words

Can rate a dropped "not" as nearly identical [2]

Text LLM-as-judge

Whether the transcript preserves what the speaker meant

Sees only the transcript, never the audio

Task-level checks, such as intent accuracy, comprehension questions or task success

Whether the downstream system can still act correctly

Needs a task setup and expected outcomes

Audio-native judges using omni or large audio models

The audio itself: meaning, tone, pronunciation, speaker

Still unreliable on some audio properties

Human evaluation

Everything, including subtle tone and context

Slow and expensive; hard to repeat


LLM judges have moved fastest. On the HATS dataset, the best LLMs agreed with human annotators on which of two transcripts was better 92–94% of the time, against 63% for WER [6].

The newer shift is to stop judging the transcript at all. Omni models such as Gemini, GPT-4o Audio and Qwen-Omni accept audio directly, so they can be asked to listen to a recording and judge it, the way a human rater would. In the AudioJudge study, audio-model judges reached a Spearman correlation of 0.902 with human preferences when ranking speech systems, and listening to the audio beat reading a transcript for tone and prosody, while transcripts still helped for pronunciation [7]. These judges are not yet a drop-in replacement. With basic prompting, their performance on paralinguistic tasks was near chance [7], and the SpeakerSleuth benchmark found that audio-language models tend to prioritize text over acoustics [8].

In practice, report WER alongside an entity error rate and at least one meaning-level score, say which normalizer you used, and keep a small human-rated set to check your automated judges against.

The words that matter most are the hardest to hear

In a support call, most words are filler. The ones that decide whether the call succeeds are phone numbers, email addresses, account and order IDs, street addresses, product names and industry terms. As AssemblyAI puts it, these words are "a small minority of tokens in any conversation and close to all of its value" [9]. A transcript can score 95% word accuracy and still get the one email address wrong, and the call fails.

The benchmarks built to test this show the gap clearly. VoiceCodeBench tests 300 human-recorded English clips containing structured values like emails, URLs and identifiers, and a clip passes only if every target value is correct [10]. Dates and plain numbers are effectively solved, but URLs, emails, file paths and postal addresses remain weak. The best model got only 62.9% of URLs right, and the best score on email addresses was 80.0%. Overall, the most accurate model passed 67.67% of clips, even though its word-level accuracy was over 94% [10].

Production logs tell the same story. A study of voice agents in telecom and retail found that speech recognition errors fell mainly on names, order IDs, phone numbers, email IDs and URLs. Phone numbers given in the requested format were frequently misheard, which led to repeated prompts and more transfers to human agents [11]. That matters to callers: in AssemblyAI's survey of voice agent users, 55% said the experience they hate most is being asked to repeat themselves [9].

Specialist vocabulary is a related problem. In healthcare, researchers found that models with low overall WER still made more errors on clinical entities such as drug names and conditions, the words that matter most for patient safety [3]. The same applies to product names in retail, plan names in telecom and policy terms in insurance.

To measure this, build a test set from your own call types, with entity labels on every phone number, email, ID and domain term. Score each value as all-or-nothing, since one wrong digit fails the task. Include the different ways people say the same thing ("john dot smith at gmail dot com", spelled-out letters, digits read in groups), and test whether your provider's custom vocabulary or keyword features actually help on your terms.

Your callers' world is louder than benchmark audio

Most ASR test sets are recorded in quiet rooms, close to the microphone, with one person talking. Real callers are on speakerphone in a car, in a busy kitchen, in an open-plan office, or on a street corner, often with someone else talking nearby. As one speech-enhancement team argues, voice agents more often fail because they hear too much than because they can't hear at all [12].

Not all noise is equal. Steady background sound, like an engine or air conditioning, mostly makes words harder to recognize. A competing voice, from a TV, a coworker or a child, causes a different and worse failure. Because background conversations carry speech-like structure, both the speech recognizer and the voice activity detector can mistake them for the caller [13]. The agent transcribes words the caller never said, treats a side conversation as a new request, or stops talking because it thinks it has been interrupted. Echo of the agent's own voice on speakerphone causes similar problems. This is also why a single signal-to-noise figure is a poor summary of noise robustness [13].

Benchmarks are starting to catch up. Hugging Face and Treble launched a Far-Field ASR Leaderboard in 2026 that tests recognition when speech competes with reverberation, background noise and other people talking [14]. At the task level, τ-Voice found that voice agents completed 31–51% of tasks with clean audio but only 26–38% with background noise and varied accents [15].

Accents add another layer, and averages hide who gets left behind. A Stanford study of five commercial speech recognition systems from Amazon, Apple, Google, IBM and Microsoft found an average WER of 0.35 for Black speakers compared with 0.19 for white speakers. The gap was just as large on identical phrases, which pointed to the acoustic models rather than vocabulary [16]. That study is from 2020, but almost no public leaderboard reports results by speaker group, so you can't tell whether the gap has closed. A more recent study of Whisper and WhisperX on medical texts found an error gap of up to 11 percentage points between native and non-native English speakers [17].

The most useful way to test this is to hold everything else fixed and change only the acoustic conditions: the same script, voice and expected outcome, replayed with an engine running, with a TV on, with another person talking, and in silence. That shows whether your agent struggles with noise in general or specifically with competing speech or accents [18]. Report results by condition and by accent group, and put the worst result next to the average.

Tone and emotion don't fit in a transcript

"Fine." means one thing said warmly and the opposite said through gritted teeth, but a transcript records the same word either way. In a pipeline agent, everything the speech recognizer doesn't write down, such as tone, hesitation, frustration and sarcasm, is gone before the language model sees the caller's turn. WER can be perfect while the agent completely misreads the call. Speech-to-speech models can hear tone, but public benchmarks for it are thin, so if emotion matters in your use case, you will mostly be building your own test set. The audio-native judges described earlier are the most promising way to score it, with the caveats noted there.

The output side has a similar blind spot. TTS arenas such as the Artificial Analysis Speech Arena rank voices by blind pairwise preference on which sample sounds more natural [19], which rewards how a voice sounds on typical sentences. That is not the same as handling hard ones: Artificial Analysis runs a separate pronunciation robustness benchmark alongside the arena, and when HeyGen Voice debuted at the top of the controlled-voice arena in October 2026, it ranked only 10th of 29 models on pronunciation robustness [20]. Nor does any public benchmark check whether the agent sounds apologetic when it should, or reads an email address back clearly enough for the caller to confirm it. A short listening test on your own scripts, including the confirmations and apologies your agent actually says, is still the most reliable check.

Benchmarks time single answers, callers experience conversations

In conversation between people, the typical gap between turns is roughly 200 ms, across many languages [21], and callers feel every few hundred milliseconds beyond that. The latency most leaderboards publish is much narrower. Artificial Analysis, for example, reports time to first audio as the average across its Big Bench Audio questions: one question, one answer, measured from the API [22].

That leaves out most of what callers notice. It skips endpointing, the wait to decide the caller has finished speaking, which is often the largest single delay in a pipeline agent. It reports an average, which hides the one turn in twenty that takes 2.5 seconds. And it runs one request at a time from a data center, while your callers are on mobile networks at peak hour. Turn-taking behavior matters as much as raw speed: Full-Duplex-Bench scores whether a model waits through a natural pause, takes the turn when appropriate, handles interruptions, and keeps going when the caller says "yeah". Yet Artificial Analysis dropped this from its speech-to-speech index in v2.0, so headline rankings no longer reflect it [22].

Measure voice-to-voice latency instead: the time from the end of the caller's speech to the first audio they hear, taken from recordings of both sides of the call and reported as p50, p95 and p99 at realistic load. Add how quickly the agent stops when interrupted, and how often it stops when it shouldn't, for a cough, a TV or a "mm-hm".

Passing the benchmark isn't finishing the task

The question callers care about is whether the agent got the job done, and component benchmarks don't ask it. The few that do show a large gap. τ-Voice gives voice agents 278 customer-service tasks that require following company policy and using tools, with simulated callers who have varied accents and realistic background audio. The best text-based model completed 85% of all tasks. Voice agents completed 31–51% of all tasks with clean audio and 26–38% with noise and accents, which is only about 30–45% of the text model's success rate. Most failures (79–90%) came from the agent's own behavior, not the test setup [15].

Strong reasoning in text does not carry over to voice, and the drop from clean to realistic audio stacks on top of it. Real-world results are likely lower still, since simulated callers are more patient and consistent than real people. This is the strongest single argument for testing whole calls rather than individual parts.

The takeaway

Public leaderboards are useful for building a shortlist, but they test clean audio, count every word equally and time single answers. Your callers bring noise, side conversations, accents, email addresses and impatience. Before you choose or ship a voice agent, test it on your own calls: score the words that carry the transaction, replay the same scenarios under realistic noise, measure how long callers wait and how often they're cut off, and track whether the task gets done. Re-run those tests whenever a vendor ships a new model, because the leaderboard won't tell you whether it works for your callers.

References

  1. Srivastav, V., et al. "Open ASR Leaderboard: Towards Reproducible and Transparent Multilingual and Long-Form Speech Recognition Evaluation." arXiv, October 8, 2025. https://arxiv.org/abs/2510.06961v2

  2. Ellis, Z., Joselowitz, J., et al. "WER is Unaware: Assessing How ASR Errors Distort Clinical Understanding in Patient Facing Dialogue." arXiv, November 20, 2025. https://arxiv.org/pdf/2511.16544v1

  3. Afonja, T., et al. "Performant ASR Models for Medical Entities in Accented Speech." arXiv, June 18, 2024. https://arxiv.org/pdf/2406.12387

  4. "Voice of India: A Large-Scale Benchmark for Real-World Speech Recognition in India." arXiv, April 21, 2026. https://arxiv.org/pdf/2604.19151

  5. Sasindran, Z., Yelchuri, H., and Prabhakar, T. V. "SeMaScore: A New Evaluation Metric for Automatic Speech Recognition Tasks." Interspeech 2024. https://arxiv.org/abs/2401.07506v2

  6. Bañeras-Roux, T., et al. "Evaluation of Automatic Speech Recognition Using Generative Large Language Models." arXiv, April 23, 2026. https://arxiv.org/abs/2604.21928

  7. Manakul, P., Gan, W. H., et al. "AudioJudge: Understanding What Works in Large Audio Model Based Speech Evaluation." Proceedings of EACL 2026, March 2026. https://preview.aclanthology.org/ingest-eacl/2026.eacl-long.168.pdf

  8. Lee, J., Pyo, J., Seo, G., and Jo, Y. "SpeakerSleuth: Evaluating Large Audio-Language Models as Judges for Multi-turn Speaker Consistency." arXiv, January 7, 2026. https://arxiv.org/abs/2601.04029v1

  9. AssemblyAI. "The voice agent accuracy problem nobody benchmarks." September 30, 2026. https://www.assemblyai.com/blog/voice-agent-accuracy-problem-benchmarks

  10. Vals AI. "VoiceCodeBench." Accessed November 17, 2026. https://vals.ai/benchmarks/voice-code-bench

  11. Singh, S., et al. "LLM-as-a-Judge for Voice-Agent Evaluation: Alignment, Stability, and Human Oversight." arXiv, 2026. https://arxiv.org/pdf/2608.24314

  12. ai-coustics. "Behind Quail Voice Focus 2.0: Technical benchmark and evaluation." March 11, 2026. https://ai-coustics.com/blog/voice-focus-2.0-deepdive

  13. Smallest.ai. "How AI Phone Agents Handle Background Noise and Poor Call Audio." October 5, 2026. https://smallest.ai/blog/how-ai-phone-agents-handle-background-noise-and-poor-call-audio

  14. Treble Technologies. "Treble Technologies and Hugging Face Address Voice AI's Unspoken Dilemma With Groundbreaking Benchmark of ASR Models." June 2026. https://treble-technologies.reportablenews.com/pr/treble-technologies-and-hugging-face-address-voice-ai-s-unspoken-dilemma-with-groundbreaking-benchmark-of-asr-models

  15. Ray, S., Dhandhania, K., Barres, V., and Narasimhan, K. "τ-Voice: Benchmarking Full-Duplex Voice Agents on Real-World Domains." arXiv, March 2026. https://arxiv.org/pdf/2603.13686

  16. Koenecke, A., et al. "Racial disparities in automated speech recognition." Proceedings of the National Academy of Sciences, April 7, 2020. https://www.pnas.org/doi/10.1073/pnas.1915768117

  17. "Accents Still Confuse AI: Systematic Errors in Speech Transcription and LLM-Based Remedies." medRxiv, 2025. https://www.medrxiv.org/content/10.1101/2025.08.29.25333548.full.pdf

  18. Coval and ai-coustics. "Voice AI Background Noise Testing: What Breaks STT." October 6, 2026. https://www.coval.ai/blog/coval-ai-coustics-background-noise-testing/

  19. Artificial Analysis. "Text to Speech Arena Leaderboard." Accessed November 17, 2026. https://artificialanalysis.ai/text-to-speech/leaderboard

  20. Artificial Analysis. "Text to Speech Benchmarking Methodology" (Pronunciation Robustness benchmark). Accessed November 20, 2026. https://artificialanalysis.ai/text-to-speech/methodology

  21. Stivers, T., et al. "Universals and cultural variation in turn-taking in conversation." Proceedings of the National Academy of Sciences, 2009. https://www.pnas.org/doi/10.1073/pnas.0903616106

  22. Artificial Analysis. "Speech to Speech Benchmarking Methodology." Accessed November 17, 2026. https://artificialanalysis.ai/methodology/speech-to-speech-benchmarking