Audio Input: Speech-to-Text vs Native Audio Models

You can send audio to a speech recogniser and the resulting text to a language model, or send the audio to a model that ingests it directly. The second is newer and usually described as simply faster. The more useful framing is that the first one deletes information, and whether you needed that information decides the choice.

Two shapes

pipeline
  mic -> VAD -> ASR -> text -> LLM -> text -> TTS -> speaker
                       ^
                       everything not in the words dies here

native audio
  mic -> audio tokens -> model -> audio or text tokens -> speaker

A native audio model discretises the waveform into tokens with a neural codec and puts them in the same sequence as text tokens — the same trick as images, applied to sound. There is no intermediate string, so nothing has to be representable as writing in order to reach the model.

What the transcript throws away

“Fine.” is not one utterance. Spoken flatly it is agreement; spoken with a rising drawl it is the opposite; spoken with a crack in the voice it is something else again. Transcription maps all three to the same four characters. What else goes:

• Prosody and emotion. Stress, pitch contour, hesitation, sarcasm. For support-call triage or anything where “how did the customer sound” is the question, this is the entire signal.
• Non-speech sound. A dog, a door, a machine whining, a cough. Some recognisers emit crude event tags; most emit nothing.
• Overlap and who spoke. Diarisation is a separate model with its own errors, and simultaneous speech usually collapses to whichever voice was louder.
• Uncertainty. The recogniser commits to one string. The alternative it nearly chose is gone, so the language model cannot reconsider a mishearing using context that arrives later in the sentence.
• Timing. How long a pause was, whether the speaker rushed, whether two people spoke at once. Some of this is recoverable from word timestamps if your recogniser emits them; most of it is not.
• Pronunciation and accent. Sometimes the thing you wanted — language identification, a name spoken but never spelled.

Against that, the transcript has enormous practical advantages, and they are the reason the pipeline is still the default: it is human readable, greppable, storable, cheap to re-query without re-sending audio, and easy to redact. A native model gives you none of those unless you also transcribe.

The cost shapes differ too, and in a direction that surprises people. Speech recognition is typically billed per minute of audio, which is a small, extremely predictable number, and the resulting transcript is a few hundred text tokens per minute of speech. Native audio input is billed as tokens, and audio tokenises at a rate on the order of tens of tokens per second — Google documents roughly 32 tokens per second for audio in the Gemini API as of August 2026. Run the multiplication before choosing: an hour of audio is a few hundred thousand tokens as audio and perhaps ten thousand as text. For anything long and re-queryable, the transcript is not just more useful, it is dramatically cheaper.

A latency budget, not a benchmark

No measurements are claimed here — the terms are listed and the values are yours to fill in from your own stack, which is the only version that will be true for your deployment.

The structural claim you can make without measuring anything: the pipeline contains two extra sequential model calls, ASR finalisation and TTS start, and sequential terms add. That is why native audio tends to win on conversational latency. It is also why a badly tuned endpointing threshold can dominate both — a 700 ms silence timeout costs more than either extra hop, and no model choice repairs it.

Error rate, and when it is the wrong metric

Word error rate is edit distance over words, normalised by reference length: WER = (S + D + I) / N, for substitutions, deletions and insertions. It is the standard number and it has two well-known problems for product work.

First, it weights every word equally. Getting “the” wrong and getting the account number wrong cost the same, which does not match any application ever built. Track entity error rate on the fields you care about — names, numbers, product SKUs — alongside WER.

Second, it does not capture the failure that hurts most. Whisper (Radford et al., 2022) is a sequence-to-sequence model trained on weakly supervised web audio, and its own paper discusses the resulting tendency to produce fluent hallucinated text — most visibly on silence or noise, where it can emit a plausible sentence with nothing behind it. A hallucinated segment in a medical or legal transcript is a different category of problem from a few wrong words, and a WER figure averaged over a clean test set will not show it. Guard with voice activity detection before the recogniser, and treat output on low-energy segments as suspect.

Choosing

• Batch, archival, searchable, auditable. Pipeline. You want the artefact, and the extra latency is irrelevant offline.
• Live conversation with a human. Native audio, if the model you want is available with it — fewer sequential hops, and interruption handling is far easier when one model owns the turn.
• The answer is in how it was said. Native audio. The pipeline has already deleted your feature.
• Regulated, redacted, or reviewed by humans. Pipeline, or native with a parallel transcript. You cannot redact a waveform you never turned into text.