Prosody, Pauses and Emphasis in Text-to-Speech

Prosody is duration, pitch and loudness — how long each sound lasts, how the fundamental frequency moves across it, and how much energy it carries. Modern text-to-speech predicts all three from the text, which is why control over them is indirect and why half the markup people write has no effect at all.

What prosody is, mechanically

Take one sentence — “I didn’t say she took the money” — and note that it has seven distinct meanings depending on which word carries the accent. Nothing about the phonemes changes. What changes is three measurable quantities:

Silence is the fourth control and the most underrated. A 250 ms pause before a phrase does more to make it land than any tag, and it is the one thing every engine supports reliably.

Where the model gets it from

Two broad designs, and they respond to control very differently.

• Explicit variance prediction. The FastSpeech 2 lineage runs the text through an encoder and then a variance adaptor that predicts duration, pitch and energy per input unit before the decoder generates the spectrogram. Because those quantities are separate predicted tensors, they can be overridden — this is the architecture where a <prosody rate="slow"> tag can be implemented as a genuine scaling of the predicted durations, and where per-word pitch control is straightforward to expose.
• Implicit, autoregressive. Codec-language-model TTS treats audio as a sequence of discrete tokens from a neural codec and generates them autoregressively, conditioned on the text and on a reference audio prompt. Prosody is not a separate variable anywhere in the model; it emerges from the token distribution. These systems sound markedly more natural and are markedly harder to control, because there is no duration tensor to scale. Control is exercised through the text, through the reference audio, or through natural-language style instructions the model was trained to follow.

The industry has moved decisively towards the second design, and the practical consequence is that the SSML controls people learned in the first era increasingly do nothing. That is not a bug report; it is a different architecture with a different control surface.

The SSML surface

Speech Synthesis Markup Language is a W3C recommendation, currently at version 1.1. These are the elements that matter for prosody, and what each is defined to do:

<speak>
  <!-- Pause. The most reliably supported control there is. -->
  Your balance is <break time="300ms"/> four hundred pounds.
  <break strength="medium"/>

  <!-- Rate, pitch and volume. Relative values travel better than
       absolute ones. -->
  <prosody rate="90%" pitch="-2st" volume="+2dB">
    spoken a little slower and lower
  </prosody>

  <!-- Emphasis. Abstract: the engine decides how to realise it. -->
  I said <emphasis level="strong">Thursday</emphasis>.

  <!-- Interpretation. This is the highest-value family and the one
       most often forgotten. -->
  <say-as interpret-as="telephone">02079460958</say-as>
  <say-as interpret-as="date" format="dmy">03/04/2026</say-as>
  <say-as interpret-as="characters">SW1A</say-as>
  <say-as interpret-as="ordinal">3</say-as>

  <!-- Pronunciation, by IPA or by substitution. -->
  <phoneme alphabet="ipa" ph="təˈmɑːtəʊ">tomato</phoneme>
  <sub alias="Multigrid A I">multigrid.ai</sub>

  <!-- Structure. Paragraph and sentence boundaries carry phrasing. -->
  <p><s>First sentence.</s><s>Second sentence.</s></p>
</speak>

Note the two st and dB units on prosody: semitones and decibels are relative and perceptually linear, so -2st means the same thing on a bass voice and a soprano one, where pitch="180Hz" does not.

Why half your tags do nothing

SSML is a specification, not a contract. Support is partial and inconsistent in ways that are rarely documented at the level of detail you need, and the honest summary is:

• Support is per voice, not per vendor. The same provider commonly has an older concatenative or variance-adaptor voice family with broad SSML support and a newer neural family that accepts a much smaller subset — sometimes only break and say-as. Reading the vendor’s SSML page tells you what the platform accepts; it does not tell you what a particular voice honours.
• Unsupported tags are usually ignored silently. You get audio back, it sounds fine, and nothing indicates that your careful emphasis markup was stripped. There is no error to catch. This is the reason the measurement section below exists.
• Some engines reject SSML entirely in favour of plain text plus a natural-language instruction describing the delivery. Neither approach is wrong; they are not interchangeable and code written for one produces nothing on the other.
• Nested and overlapping tags are where support ends first. A prosody inside an emphasis inside a voice is well-formed XML and frequently unimplemented.

Measuring whether a tag changed anything

You do not need a listening test to answer “did this tag do anything at all”. Two properties of the returned audio settle it, and both come out of ffmpeg with no audio library involved: total duration, and where the silences are.

#!/usr/bin/env bash
# ssml-probe.sh -- does this voice honour this tag?
#
# Usage: put your vendor's synthesis call in synth(), then run.
# Compares each variant against the plain-text baseline on
# duration and on pause structure.

set -euo pipefail

synth() {   # $1 = text or SSML, $2 = output wav
  # REPLACE THIS with your provider's call. It must write a wav
  # to $2. Nothing else in this script is vendor-specific.
  : "$1" "$2"
  false
}

probe() {   # $1 = label, $2 = input
  local wav="out-$1.wav"
  synth "$2" "$wav"

  local dur
  dur=$(ffprobe -v error -show_entries format=duration \
        -of default=nw=1:nk=1 "$wav")

  # Silences longer than 150 ms, at least 35 dB below peak.
  local pauses
  pauses=$(ffmpeg -v info -i "$wav" \
           -af silencedetect=n=-35dB:d=0.15 -f null - 2>&1 \
           | grep -c silence_start || true)

  printf '%-12s duration=%-8s pauses=%s\n' "$1" "$dur" "$pauses"
}

TEXT='Your balance is four hundred pounds.'

probe baseline  "$TEXT"
probe break     "<speak>Your balance is <break time=\"600ms\"/> four hundred pounds.</speak>"
probe slow      "<speak><prosody rate=\"70%\">$TEXT</prosody></speak>"
probe emphasis  "<speak>Your balance is <emphasis level=\"strong\">four hundred</emphasis> pounds.</speak>"

Read the output like this. If break is honoured, the break row has one more detected pause than baseline and its duration is roughly 600 ms longer. If rate is honoured, the slow row is substantially longer with the same pause count — at 70%, expect something near 1.4 times the baseline duration. If either row is within a few tens of milliseconds of baseline, the tag was ignored, and no amount of listening will change that.

emphasis is the one this cannot settle. Emphasis is realised mostly in the pitch contour, which duration does not reveal. For that, extract an F0 track — librosa.pyin in Python, or Praat if you have it — and compare the semitone range over the emphasised words against the baseline. If the contour is identical, the tag did nothing.

Run this once per voice you ship and keep the results next to the voice configuration. It takes twenty minutes and it replaces an argument that otherwise recurs every time somebody joins the team.

What works in practice

1. Punctuate the text properly first. In neural TTS, punctuation is a stronger and far more reliable prosodic control than most markup, because the model was trained on punctuated text and learned the correspondence directly. A full stop instead of a comma changes the phrase boundary, the final lengthening and the pitch reset. Fixing punctuation on generated text — see punctuation restoration — improves synthesis as a side effect.
2. Rewrite rather than mark up. “It is four hundred pounds. Four hundred.” produces the emphasis you wanted on every engine, forever, with no tag. Splitting a long sentence in two does more for intelligibility than any rate setting.
3. Use say-as for everything that is not a word. Phone numbers, postcodes, reference numbers, dates, currency, ordinals. This is where synthesis most often produces something a caller cannot act on, and it is the best-supported part of SSML.
4. Insert pauses at decision points. Before a number the caller must write down; after a question, so the pause is part of the audio rather than an accident of your turn logic. On a phone agent this doubles as a chance for the caller to interrupt.
5. Slow down numbers, not prose. A global rate reduction sounds patronising over a whole utterance and is genuinely helpful over an account number.
6. Verify pronunciation of your own vocabulary. Product names, place names and industry abbreviations are where a voice embarrasses you. Fix them once with sub or phoneme and keep a lexicon file in the repository.