Skip to content
Multigrid

Speech, audio and voice engineering

How speech becomes tokens and tokens become speech, and the latency and cost arithmetic that decides whether a voice product feels human.

Speech is the one interface where the machine is judged against a standard nobody agreed to. People expect a reply to land in the gap they leave for a reply, and that gap is short — a few hundred milliseconds in ordinary conversation. Miss it and the caller does not think the system is slow, they think it is not listening.

That constraint is what makes voice engineering arithmetic rather than taste. Every hop in the audio path spends milliseconds, and the sum is what the caller experiences. So most of these pages add something up: the path from microphone to speaker, the cost of a minute of call, the silence you wait for before deciding somebody has finished, the extra hop that translation inserts. Every term is named and left as a symbol until a line that is clearly marked as illustrative.

The rest is the machinery on either side — how a spectrogram becomes tokens, why the same transcript scores 75% word error rate or 10% depending only on how you normalised it, what a cloned voice does and does not copy, and where recording law makes an engineering decision for you.

How Speech Recognition Works Now

The full path from microphone samples to text: framing, mel spectrograms, an encoder, and the three decoder families that behave completely differently.

10 min read

Whisper's Architecture and Its Known Failure Modes

The 30-second window, the multitask token sequence, and the mechanical reason Whisper writes captions over silence.

10 min read

Word Error Rate: Measuring Transcription Properly

WER computed by hand on one sentence pair, then the same pair scored four ways to show that normalisation moves the number from 75% to 10%.

10 min read

Speaker Diarisation: Who Said What

The embed-and-cluster pipeline, the collar and overlap decisions that move diarisation error rate more than the model does, and when not to diarise at all.

9 min read

Voice Activity Detection and Endpointing

How VAD works, and the arithmetic behind the silence threshold that decides whether your voice agent interrupts people or leaves them waiting.

9 min read

Streaming Transcription and Stable Partial Text

How to send audio over a WebSocket and render revising partial hypotheses without the flicker, with a stable-prefix algorithm you can paste in.

11 min read

Prosody, Pauses and Emphasis in Text-to-Speech

What a TTS model predicts when it decides how a sentence sounds, which SSML tags reach that decision, and a measurement protocol for finding out on your own voice.

10 min read

Voice Cloning: How It Works and the Consent Question

The speaker-embedding and codec-language-model routes to cloning a voice, what a clone does not copy, and a consent record built as an engineering artefact.

10 min read

Building a Phone Agent: SIP, WebRTC and the Audio Path

Every hop between the caller's mouth and their ear, named and added up as a symbolic budget you refill with your own measurements.

13 min read

Barge-In: Letting a Caller Interrupt

Echo cancellation, the half-duplex trap, the audio already in flight when you stop, and a state machine that handles all three.

11 min read

Audio Preprocessing That Improves Accuracy

The format conversions that are mandatory, the enhancements that are arguable, and a harness that scores every variant against your own audio.

11 min read

Accents, Code-Switching and Multilingual Calls

Why recognition accuracy varies by speaker group, what published evaluation has actually shown, and how to measure the gap in your own traffic.

10 min read

Timestamps, Alignment and a Searchable Archive

Where word timestamps come from, why some of them are guesses, and a full-text index that jumps from a search hit to the second it was said.

11 min read

Punctuation, Casing and a Readable Transcript

The post-processing that turns recogniser output into publishable text, and the verification step that stops a language model quietly rewriting what was said.

10 min read

Real-Time Speech Translation

Cascade against direct models, the latency budget for each, and the word-order problem that makes simultaneous translation structurally harder than translation.

10 min read

Audio Fingerprinting and Deduplicating a Media Library

How a ten-second clip finds its source track: spectral peaks, combinatorial hashing, and the storage arithmetic that decides whether your index fits.

10 min read

Sound Event Detection Beyond Speech

Tagging, detection and scene classification on non-speech audio, and the false-positive arithmetic that decides whether a continuous monitor is usable.

9 min read

Recording, Consent and Retention for Voice Products

The shape of call recording and retention rules, the axes on which jurisdictions differ, and the questions to settle with counsel before you build.

10 min read

Audio Quality Metrics That Are Not "Sounds Good"

MOS and the standards behind it, the intrusive and non-intrusive automatic metrics, and how many listeners a comparison needs before a difference means anything.

10 min read

Costing a Voice Product Per Minute

ASR plus model plus TTS plus telephony, added up as an equation with prices left as symbols, plus the quadratic context term that makes long calls disproportionately expensive.

12 min read

Other topics