Accents, Code-Switching and Multilingual Calls

Speech recognition does not fail uniformly. It fails more for some speakers than others, in ways that a single corpus-level word error rate hides completely — and the only number that describes your callers is one you compute yourself.

Why the gap exists

Three mechanisms, and they compound. None of them is mysterious and none is fixed by a larger model alone.

• Acoustic mismatch. The encoder maps spectral patterns to units it learned from training audio. A vowel realised differently from the training distribution lands in a thinner region of the model’s experience, and the representation is correspondingly less discriminative. This is a straightforward distribution-shift argument and it applies equally to regional accents, second-language speech, age, and speech affected by illness or a disability.
• The language model prior pulls towards the frequent form. Every recogniser with an internal or external language model resolves acoustic ambiguity by preferring likely word sequences. When the acoustics are already less certain, the prior carries more of the decision — so ambiguous input is resolved towards whatever was common in the text the model saw. Names, places and dialect vocabulary are exactly what that prior is weakest on, and exactly what is most likely to appear in the speech of the groups already disadvantaged acoustically.
• The metric penalises legitimate variation. A speaker who says something dialectally correct that differs from the reference transcript’s convention is scored as wrong. Some of the measured gap is a recognition failure and some is a normalisation decision about what counts as the right transcript, and the two are routinely reported as one number.

What published evaluation has shown

The most-cited result is Koenecke and colleagues, published in the Proceedings of the National Academy of Sciences in 2020, which evaluated five commercial speech recognition systems on matched interview recordings and found average word error rates roughly twice as high for Black speakers as for white speakers — in the region of 0.35 against 0.19 — with the gap persisting after controlling for the content of what was said.

The other genuinely useful outputs of that literature are the corpora, because they are what make a measurement possible at all:

None of these describes your callers. They are instruments for comparing models before you choose one; the number that matters comes from your own audio.

Code-switching is a different problem

Accent variation is one language realised differently. Code-switching is two languages in one utterance, and it breaks a structural assumption rather than a statistical one.

Most recognisers take a single language for the whole request — Whisper emits one language token per window and decodes the rest under it. An utterance that switches languages mid-sentence has no correct value for that token. The system does one of three things, all of them bad: transcribes the second language phonetically in the first language’s orthography, silently translates it, or drops it.

Intra-sentential switching — a Hindi sentence with English technical nouns, a Spanish sentence with an English brand name — is the common case in real customer conversations and the hardest one. Practical approaches, in the order most people should try them:

1. Check whether your provider has a multilingual or code-switching mode. Several do, under names that vary, and it is a configuration change rather than a project. Confirm it works on your audio rather than on the demo.
2. Segment first, then recognise. Run language identification per utterance or per speech region and route each to the appropriate configuration. This handles inter-sentential switching well and intra-sentential switching not at all.
3. Bias the vocabulary. If the switched-in words are a known finite set — product names, technical terms, place names — phrase biasing gets you most of the benefit for a fraction of the work. See below.
4. Fine-tune on switched audio. Effective and expensive, and it needs transcribed in-domain data you probably have to create.

Measuring your own gap

This is the part that is actually actionable, and it is a week of work rather than a research project.

1. Define cohorts by something you can observe and defend. Self-reported language preference, the country of the phone number, the region of the service, the channel. Do not infer sensitive characteristics from audio in order to measure them — that creates a new problem while measuring an old one, and it is precisely the kind of processing that attracts obligations under most privacy regimes. Where the cohort you care about cannot be observed lawfully and proportionately, say so and measure the proxies you can.
2. Sample within cohort, not across. Fifty to a hundred utterances per cohort, drawn from real traffic rather than from the calls somebody remembered. Stratify by recording condition too, so you are not comparing one cohort’s mobile calls with another’s headsets.
3. Transcribe references by hand, with one written convention document covering numbers, fillers, false starts and dialect forms. Two annotators on a subset, and check they agree before trusting either.
4. Score with one normaliser across all cohorts, and report S, D and I separately. A cohort with high deletions is being heard less; a cohort with high substitutions is being misheard. These need different fixes.
5. Put a confidence interval on every cohort. With fifty utterances, the interval is wide. Bootstrap over utterances, report the interval, and do not act on a difference whose intervals overlap heavily. The most common error in this work is a reorganised product built on noise.
6. Re-run it on every model change. An upgrade that improves the aggregate can widen a gap. Aggregate WER cannot show you that, which is the whole reason for the exercise.

The harness in audio preprocessing runs this unchanged: point it at one cohort’s directory at a time.

What actually helps

• Phrase biasing, if your provider supports it. Supplying the terms that matter — product names, place names, the fifty items in your catalogue — raises their probability during decoding, and it is the cheapest large win available for names and technical vocabulary regardless of accent. The parameter is called different things by different vendors, so check the docs; the concept is universal.
• Set the language explicitly. Automatic detection is one prediction made on the worst audio in the call, and getting it wrong costs you the entire transcript rather than a few words.
• Fix the audio path before blaming the model. Narrowband telephony removes the high-frequency energy that distinguishes fricatives, and the resulting confusions look exactly like accent effects in an error analysis. Rule that out first; see the codec table.
• Evaluate more than one recogniser on your cohorts. Models differ substantially in which speakers they handle well, and the aggregate leaderboard does not predict it. If one cohort is badly served, a different model for that route is a legitimate answer.
• Fine-tune on in-domain audio when you have hundreds of hours and the gap persists. Effective, and the last thing to reach for because everything above is cheaper.

Product decisions that follow

Some of the gap will not close, and the product has to be honest about that rather than pretending otherwise.

• Always offer a route out. A caller the recogniser cannot understand must be able to reach a person or a keypad without fighting for it. Repeated recognition failure is the strongest signal you have that this caller needs a different path, and it is trivially available: count consecutive low-confidence turns and escalate.
• Confirm rather than assume on high-cost fields. Names, addresses, account numbers, amounts. Read back and confirm, for everybody — a confirmation step applied only to speakers the system judged uncertain is both worse engineering and worse treatment.
• Never surface a raw confidence score to the caller, and never let a low score change the tone of the reply. The system being unsure is not the caller’s fault and should not be presented as it.
• Publish the cohort table internally. A gap that appears in a dashboard gets fixed; one that lives in an aggregate never surfaces at all.