Audio Preprocessing That Improves Accuracy

Some audio preprocessing is a format requirement and always correct. Some is an enhancement that sounds better to a human and can make recognition worse. Nobody can tell you which is which for your audio, so the second half of this page is a harness that finds out.

The conversions that are not optional

These are not improvements, they are contract compliance. Getting one wrong produces silently degraded output rather than an error.

# The canonical conversion. soxr is a high-quality resampler.
ffmpeg -i input.m4a \
       -ar 16000 -ac 1 -c:a pcm_s16le \
       -af "aresample=resampler=soxr:precision=28,highpass=f=60" \
       output.wav

# Check what you actually have, before and after:
ffprobe -v error -show_entries \
  stream=sample_rate,channels,bits_per_raw_sample,codec_name \
  -of default=nw=1 output.wav

Also check for clipping before anything else. A recording whose samples are pinned at full scale has lost information that no processing recovers, and the distortion it introduces is broadband — exactly the kind of corruption a spectrogram front end handles worst.

# Fraction of samples at or near full scale. Anything above a
# fraction of a percent means the recording chain needs fixing,
# not the file.
ffmpeg -i input.wav -af astats=metadata=1 -f null - 2>&1 \
  | grep -E "Peak level|Flat factor|Number of clipped"

The enhancements that are arguable

Everything below improves how audio sounds to a person. Whether it improves recognition depends on the model and on your audio, and the direction is genuinely not obvious.

• Loudness normalisation. Bringing every file to a consistent integrated loudness — the EBU R128 standard, implemented by ffmpeg’s loudnorm filter — removes a source of variation the model was probably not trained to expect. It is the safest of the enhancements because it is close to a linear gain change. The two-pass form measures first and then applies, which avoids the dynamic behaviour of the single-pass version.
• Noise reduction. The genuinely contentious one. Spectral subtraction and its neural successors remove noise by attenuating time-frequency bins judged to be noise, and they necessarily attenuate speech in those bins too. The residue is musical noise and smeared consonants — a signal that sounds cleaner and contains less. Modern ASR encoders were trained on large volumes of noisy real-world audio and are frequently more robust to the original noise than to the artefacts of removing it. Treat any claim that denoising helps as a hypothesis about your data, and test it.
• Dereverberation. Reverberation genuinely hurts recognition, because it smears the temporal structure the model needs and each reflection is a delayed copy of the signal. Unlike noise reduction, dereverberation attacks a distortion that the front end cannot ignore. It is the enhancement most likely to help on room recordings, and it does nothing on a phone call.
• Silence trimming. Removing long silences reduces audio-minute charges directly and removes the input condition that triggers hallucination on silence. Keep a few hundred milliseconds of padding around each speech region; trimming tight to the VAD boundary clips word onsets and costs you accuracy for the sake of a few kilobytes.
• Bandwidth extension. Neural upsampling that invents the missing high band of telephony audio. It is generative — the invented content is plausible rather than true — and feeding invented spectral detail to a recogniser is a gamble in a way that a linear resample is not. If you try it, it belongs in the harness, not in the pipeline.

The harness

Twenty to fifty files, hand-corrected references, one script. This is an evening’s work and it settles arguments that otherwise recur for years.

# sweep.py -- score preprocessing variants on your own audio.
#
# Layout:
#   audio/clip001.wav ... audio/clipNNN.wav      original recordings
#   refs/clip001.txt  ... refs/clipNNN.txt       hand-corrected truth
#
# Needs: ffmpeg on PATH, and wer.py from the word error rate page
# in the same directory.

import subprocess, pathlib, statistics, csv, sys
from wer import score, normalise

VARIANTS = {
    # name        : ffmpeg -af filter chain (empty = format conversion only)
    "baseline"    : "",
    "highpass"    : "highpass=f=60",
    "loudnorm"    : "loudnorm=I=-16:TP=-1.5:LRA=11",
    "hp_loudnorm" : "highpass=f=60,loudnorm=I=-16:TP=-1.5:LRA=11",
    "denoise"     : "afftdn=nr=12:nf=-25",
    "denoise_hard": "afftdn=nr=24:nf=-20",
    "trim"        : "silenceremove=start_periods=1:start_silence=0.2:"
                    "start_threshold=-40dB:stop_periods=-1:"
                    "stop_silence=0.4:stop_threshold=-40dB",
}

def prepare(src: pathlib.Path, dst: pathlib.Path, chain: str) -> None:
    af = "aresample=resampler=soxr:precision=28"
    if chain:
        af = af + "," + chain
    subprocess.run(
        ["ffmpeg", "-nostdin", "-y", "-v", "error", "-i", str(src),
         "-ar", "16000", "-ac", "1", "-c:a", "pcm_s16le",
         "-af", af, str(dst)],
        check=True,
    )

def transcribe(path: pathlib.Path) -> str:
    """REPLACE ME.

    Call whatever recogniser you are evaluating and return its text.
    This is the only vendor-specific line in the file. Keep every
    decoding option fixed across variants -- if you change the model
    or its settings between runs, the comparison means nothing.
    """
    raise NotImplementedError

def main(audio_dir="audio", ref_dir="refs", work="work"):
    audio = sorted(pathlib.Path(audio_dir).glob("*.wav"))
    if not audio:
        sys.exit("no audio found")
    pathlib.Path(work).mkdir(exist_ok=True)

    rows = []
    for name, chain in VARIANTS.items():
        wers, subs, dels, inss = [], 0, 0, 0
        for src in audio:
            ref_path = pathlib.Path(ref_dir) / (src.stem + ".txt")
            if not ref_path.exists():
                continue
            dst = pathlib.Path(work) / f"{name}-{src.stem}.wav"
            prepare(src, dst, chain)
            hyp = transcribe(dst)
            ref = ref_path.read_text(encoding="utf-8")
            r = score(normalise(ref), normalise(hyp))
            wers.append(r.wer)
            subs += r.substitutions
            dels += r.deletions
            inss += r.insertions

        rows.append({
            "variant": name,
            "files": len(wers),
            "mean_wer": round(statistics.mean(wers) * 100, 2),
            "median_wer": round(statistics.median(wers) * 100, 2),
            "worst_wer": round(max(wers) * 100, 2),
            "S": subs, "D": dels, "I": inss,
        })

    with open("sweep.csv", "w", newline="", encoding="utf-8") as fh:
        w = csv.DictWriter(fh, fieldnames=list(rows[0].keys()))
        w.writeheader()
        w.writerows(rows)

    for row in sorted(rows, key=lambda r: r["median_wer"]):
        print(row)

if __name__ == "__main__":
    main()

Everything vendor-specific is in one function. Everything else — the conversions, the scoring, the normalisation — is code you can read.

Reading the results

1. Look at the median, not the mean. One file where the recogniser produced nothing drags a mean WER past anything meaningful. Report both, and look at worst_wer separately: the variant that helps the median and destroys the worst case is usually the wrong choice for production.
2. Read S, D and I separately. Aggressive denoising characteristically raises deletions — it removes quiet words along with the noise. If denoise_hard lowers WER but raises D, it is not helping, it is making the recogniser say less.
3. Distrust a small win. With thirty files, a difference of half a percentage point is noise. If you want a decision you can defend, bootstrap: resample the file-level WERs with replacement a few thousand times and look at whether the difference between two variants keeps its sign.
4. Split by recording condition. The right answer is usually different for mobile calls, headset calls and room recordings. One pipeline for all of them is a decision to be wrong on two of the three.
5. Re-run when the model changes. This is a property of the pair, not of the audio. A new recogniser version can reverse the ordering, which is exactly why the harness is worth more than a conclusion.

Fixing it upstream instead

Preprocessing is repair work. Every one of these is worth more than any filter chain:

• Take the highest-quality leg available. A wideband or full-band stream beats 8 kHz telephony by more than any processing recovers. If a caller is in a browser, do not let anything in your stack downsample them to narrowband for consistency with the phone path. Consistency is not worth an accuracy ceiling; see the codec table in the audio path.
• Avoid double transcoding. Each lossy re-encode compounds. Ask your provider what the recording is encoded as and whether it was transcoded on the way.
• Record per-participant tracks. Cheaper and more accurate than any diarisation you can buy afterwards.
• Give people headsets. The single largest quality improvement available in a call centre, and it makes barge-in work as well.
• Capture at 16 kHz or above from the start. If you control the client, there is no reason to record narrowband and then upsample.