Word error rate is a bad way to choose an ASR model
Every ASR comparison you will read is organised around word error rate. Model A gets 4.1% on some benchmark, model B gets 5.3%, and the conclusion writes itself. It is a legitimate research metric and a bad procurement metric, because it answers a question almost no product is actually asking.
WER counts edits — substitutions, insertions, deletions — and divides by the number of reference words. Every word counts once, and every word counts the same. That last property is the problem. In a transcript that matters, the words are not interchangeable.
Consider a support call transcript with two errors:
"I'd like to cancel order A-4471" → "I'd like to cancel order A-4417"
"I wanna cancel my order" → "I want to cancel my order"
Both are one substitution. Both move WER by exactly the same amount. One of them routes a refund to the wrong purchase, and the other is a normalisation choice that no downstream consumer will ever notice. Any metric that scores those identically is not measuring what your product cares about.
Where the errors actually land
The failures that cost you are concentrated in a small set of token types, and they are precisely the types most likely to be out of distribution for a general model:
Proper nouns. Product names, company names, people's names. These carry most of the information in a business transcript and they are the hardest tokens in the vocabulary, because they are rare, often not real words, and frequently ambiguous even for a human listener with context.
Numbers and identifiers. Order IDs, dosages, prices, dates, account numbers. Errors here are not degradations, they are wrong facts. A model that transcribes "fifteen" as "fifty" has produced a fluent, plausible, completely wrong sentence — and unlike a garbled proper noun, nothing downstream will flag it as suspicious.
Domain vocabulary. Every industry has a few hundred words that a general-purpose model has barely seen. Get them wrong and the transcript reads as nonsense to the person who knows the field, no matter how good the WER is on the surrounding filler.
Meanwhile the words that are easy to get right — articles, fillers, discourse particles — are also the most numerous. They dominate the denominator. A model can be visibly better at "the" and "um" and "you know" than a competitor, win on WER, and lose on every transcript anyone reads.
The failure modes WER cannot see at all
Beyond the weighting problem, several product-critical behaviours are invisible to the metric entirely.
Fluent hallucination on non-speech. Modern attention-based ASR models are trained to produce plausible text, and when handed silence, breathing, music, or hold tone, some of them produce a plausible sentence anyway. Common outputs are boilerplate the model saw a lot in training — subtitle credits, "thanks for watching", stock phrases. On a benchmark with clean segmented speech this never happens, so it never shows up in the number. In a call recording with thirty seconds of hold music, it happens constantly, and it is worse than a deletion because it is confidently formatted as content.
Unstable partials in streaming. In a streaming setting, the model emits hypotheses before it has heard the rest of the sentence, and revises them as context arrives. The final transcript may be excellent — WER is computed on the final transcript — while the live caption on screen visibly rewrites itself three times per sentence. Users experience the revisions, not the final text. Two models with identical final-transcript WER can feel wildly different here, and the metric is structurally incapable of telling you which is which.
Segmentation and turn attribution. If the model merges two speakers' turns, or splits one sentence across two segments, WER on the concatenated text can be nearly perfect while the structured output your application consumes is wrong. Anything that feeds an LLM downstream cares much more about this than about the word count.
Punctuation and casing. Usually stripped before scoring by convention. Also usually the difference between a transcript a human can read and one they cannot, and — if you feed transcripts to a model — a meaningful influence on how the downstream model interprets the text.
Code-switching. Benchmarks are monolingual. Real speech in most of the world is not. Most markets outside the US speak sentences with English product names, brand names, and technical terms dropped into them, and a model that handles those switches cleanly will beat a nominally stronger one on your actual traffic. No standard benchmark will have told you which is which.
What to measure instead
None of this argues for having no metric. It argues for measuring the errors weighted by what they cost you.
Entity error rate. Take the tokens your application actually consumes — names, IDs, numbers, domain terms — and compute accuracy on those alone. This is usually a small tagging effort on a few hundred utterances and it discriminates between models far more sharply than WER does, because it removes the filler words that were flattening the difference.
Hallucination rate on non-speech. Build a small adversarial set: silence, hold music, background conversation, keyboard noise, a cough. The correct output is empty. Measure how often each model invents text. Some otherwise excellent models fail this badly, and you want to know before production rather than after.
Partial stability. For streaming, log every intermediate hypothesis and count how many times an already-displayed prefix gets rewritten per utterance. This is the number that predicts whether live captions feel solid or twitchy. Pair it with the latency at which partials are emitted, because the two trade off directly — a model can always look stable by waiting longer.
End-to-end task accuracy. If ASR feeds an LLM that extracts a structured result, score the structured result. This is the only measurement that weights errors by their true downstream cost automatically, and it will sometimes tell you that a worse transcript produces better outcomes, because the LLM repairs certain error classes and not others.
Build the test set from your own traffic
The most valuable thing here is not any particular metric — it is that the evaluation audio comes from your product. A hundred real recordings, transcribed carefully by hand, will tell you more about which model to buy than any public benchmark, because they carry your acoustics, your accents, your vocabulary, your background noise, and your microphone quality.
Public benchmarks are largely read speech or curated broadcast audio: clean, close-mic, one speaker, no cross-talk, no compression artifacts, no one calling from a car. Every one of those gaps is a place where model rankings reorder. It is routine for the benchmark winner to lose on real traffic, and the reason is almost never mysterious — it is that the benchmark did not contain the conditions your users are in.
A hundred utterances is enough to see large differences and it is a day of work. Doing it once, and rerunning it whenever you consider switching models, is the single highest-leverage evaluation practice available for speech.
The takeaway
WER is fine for tracking research progress on a fixed corpus. It is a poor basis for a purchasing decision, because it weights every word equally, ignores everything that happens before the final transcript, and cannot see the failure modes that non-speech audio and streaming introduce.
Score the tokens that carry the meaning, test explicitly for invented text, measure how much your live captions rewrite themselves, and do all of it on recordings from your own users. The ranking you get will not be the one on the leaderboard, and it will be the one that holds up.