Audio Quality Metrics That Are Not "Sounds Good"

Mean opinion score is the standard for speech quality and it is a testing procedure, not a property of an audio file. Most of what goes wrong in speech evaluation is quoting the number without the procedure — or without enough listeners for it to mean anything.

MOS is a protocol, not a number

The absolute category rating procedure is specified in ITU-T Recommendation P.800: listeners hear a sample in isolation and rate it on a five-point scale from bad to excellent, and the mean of those ratings is the MOS. P.808 extends the methodology to crowdsourced listening, which is how nearly all of it is now done. Everything about the setup is part of the measurement:

• What was rated. Overall quality, naturalness, intelligibility and speaker similarity are different questions and produce different numbers from the same audio. A MOS with no stated question is uninterpretable.
• Who listened, and on what. Naive listeners and experts diverge. Headphones, laptop speakers and a phone diverge more. Crowdsourced protocols include headphone-screening and attention-check items for exactly this reason.
• What else was in the set. Absolute ratings are anchored by context: the same sample scores higher among poor samples than among good ones. Including a natural-speech anchor and a deliberately degraded anchor in every session is the standard correction and it is frequently skipped.
• Which sentences. Text material dominates. Long sentences, unusual names, numbers and questions all stress a synthesiser differently, and a favourable sentence set is the easiest way to produce a favourable MOS.

Two variants exist because they answer better questions. Comparison MOS asks listeners to rate B relative to A on a scale from much worse to much better, which removes most between-listener variation. MUSHRA (ITU-R BS.1534) presents all systems simultaneously with a hidden reference and an anchor, and is far more sensitive to small differences — at the cost of needing trained listeners.

How many listeners you need

This is the arithmetic that makes most published quality claims evaporate, and it needs nothing but a standard error.

Assumption you must state: the per-rating standard deviation on
a 5-point scale. In speech listening tests it is typically in the
region of 0.8 to 1.0. Take sigma = 1.0 and substitute your own
measured value.

Confidence interval on ONE system's MOS, n independent ratings:

  half-width = 1.96 * sigma / sqrt(n)

    n =   25   ->  1.96 * 1.0 / 5    = +/- 0.39
    n =  100   ->  1.96 * 1.0 / 10   = +/- 0.20
    n =  400   ->  1.96 * 1.0 / 20   = +/- 0.10

Comparing TWO systems is harder, because the difference carries
both variances:

  half-width on the difference = 1.96 * sigma * sqrt(2/n)

    n =  100   ->  1.96 * 1.0 * 0.1414  = +/- 0.28
    n =  400   ->  1.96 * 1.0 * 0.0707  = +/- 0.14

To resolve a difference of 0.1 MOS between two systems:

  1.96 * 1.0 * sqrt(2/n)  <=  0.1
  sqrt(2/n)               <=  0.051
  2/n                     <=  0.0026
  n                       >=  770 ratings per system

So a paper or a landing page reporting "4.31 against 4.22" from
40 listeners rating 10 sentences each has an interval several
times wider than the difference it is claiming.

The way out is the PAIRED design. Ask each listener to compare
A and B on the same sentence, and the between-listener variance
-- the largest component -- cancels. A comparison MOS study
resolves the same difference with a small fraction of the
ratings, which is why it is the right instrument for "is the new
voice better than the old one".

Intrusive metrics, and their design scope

Intrusive metrics compare degraded audio against a clean reference of the same utterance. They are cheap, deterministic and repeatable, and their limits are specific rather than vague.

The limitation to internalise: PESQ and POLQA were built and validated to predict how a person rates a transmission of real speech. Synthetic speech is not a degraded transmission of anything. It differs from the reference in duration, in prosody and in the identity of the voice, none of which those models were designed to handle, and a time-aligned comparison against a human recording of the same sentence penalises a synthesiser for making legitimate prosodic choices. Using them to rank text-to-speech systems is applying an instrument outside the range it was calibrated for. They remain entirely appropriate for what they were built for — which, for a voice product, is the telephony path rather than the synthesiser.

MCD has the opposite problem: it is well defined for synthesis, and it measures similarity to one particular recording rather than quality. A system that sounds better than the reference scores worse.

Non-intrusive metrics

These predict a quality score from the audio alone, with no reference — which is the only option for output that has no reference, meaning nearly all generated speech.

• DNSMOS, from Microsoft, is a model trained to predict subjective ratings collected under the ITU-T P.835 methodology, which separates speech quality, background noise intrusiveness and overall quality into three scores. It came out of the Deep Noise Suppression challenge series and is aimed at enhancement systems.
• NISQA, from TU Berlin, predicts overall speech quality along with interpretable dimensions — noisiness, discontinuity, coloration, loudness — which makes it more useful for diagnosis than a single number.
• UTMOS and its relatives came out of the VoiceMOS Challenge and are trained specifically to predict listener ratings of synthesised speech, which is the closest thing available to an automatic MOS for TTS.

The honest caveat, and the reason no correlation figure appears on this page: every one of these is a model fitted to particular listening-test data, and its agreement with human ratings is reported per dataset in the paper that introduced it. Those figures do not transfer to your systems, your voices or your material, and a correlation quoted out of its dataset is not information. Use non-intrusive metrics as regression detectors — a change of this size in this metric on this test set is worth a listen — and confirm any decision with people.

Intelligibility is separate

Quality and intelligibility are different axes and a voice product usually cares more about the second. A slightly robotic voice that reads an account number correctly beats a beautiful one that does not.

• Recognition as a proxy. Run your synthesised audio through a speech recogniser and compute word error rate against the input text. It is fully automatic, it scales to every utterance you ship, and it catches the failures that matter most: mangled numbers, dropped words, wrong pronunciations of names. Its weakness is that a recogniser is not a person and will happily understand speech a human finds unpleasant — which is exactly why you also need a quality measure.
• Structured human intelligibility tests exist for the cases where this matters most, using semantically unpredictable sentences so listeners cannot infer words from context. Worth the effort for accessibility products and safety announcements.
• STOI, the short-time objective intelligibility measure, is the standard automatic intelligibility metric for enhancement and separation work. Like PESQ it needs a reference and was designed for processed natural speech.
• Speaker similarity, for cloned voices, is measured with a speaker verification model: embed the reference and the synthesis and take the cosine similarity. This is an objective measure of the thing cloning claims to do, and it is independent of whether the result sounds good.

A workable evaluation

1. Fix a test set and never change it silently. A few hundred utterances covering your real material — the numbers, names, addresses and questions your product actually says, not a poetry corpus. Version it.
2. Run automatic checks on every build. Recognition WER for intelligibility, a non-intrusive predictor for quality, speaker similarity if you are cloning, and total duration for pacing regressions. Alert on change rather than on absolute value.
3. Use paired human comparison for decisions. When choosing between two voices or two vendors, run a comparison MOS with the same sentences and randomised presentation order. It answers the actual question with an achievable number of ratings.
4. Test through the delivery path. Evaluating a 48 kHz studio render of a voice that will be heard through 8 kHz telephony measures something nobody will experience. Encode the test audio through your real codec chain first; it changes the ranking more often than people expect.
5. Keep the failures. Every utterance a listener flags or a recogniser mangles goes into a regression set. That set is worth more after a year than any metric, because it is the only record of what has actually gone wrong in your product.