Real-Time Speech Translation

Speech translation adds one hop to the voice pipeline and one problem that is not about latency at all: in some language pairs the information needed to translate the beginning of a sentence does not arrive until the end of it.

Two architectures

There is also a partial case worth knowing: Whisper’s translate task is direct speech-to-text translation, but only into English. If your target is English it is a single hop; if it is anything else you are cascading regardless.

For most products the cascade wins on operational grounds rather than quality grounds. You can see the intermediate transcript, log it, show it to a human, correct it, and swap any one component without touching the others. Direct models win where the intermediate text genuinely loses something — preserving the speaker’s voice through the translation is the clearest case.

The latency budget, both ways

Take the audio-path budget from the phone agent page and substitute the processing terms. Everything network-side is unchanged.

CASCADE

  gap = D_in + e + r + t + s + D_out

    D_in, D_out  network terms, unchanged from any voice pipeline
    e            endpoint wait (or the segmentation decision, if you
                 are not waiting for a turn to end)
    r            ASR finalisation
    t            translation: time to first target token
    s            TTS time to first audio

DIRECT (speech to speech)

  gap = D_in + e + u + D_out

    u            one model, source audio in, target audio out

The direct form removes two handoffs and one finalisation. It does
NOT remove e, and it does not remove the lag below.

What overlaps:

  r with t   issue translation on the ASR partial; reissue if the
             final differs
  t with s   stream target tokens into TTS at clause boundaries
  neither    e, and the read-write lag L

So the realistic cascade figure is:

  gap ~= D_in + e + L + max(r, t_first) + s_first + D_out

and the honest observation is that L -- discussed next -- is
frequently larger than every processing term put together.

Why waiting is unavoidable

Consecutive translation waits for the speaker to finish. Simultaneous translation emits target words while the source is still being spoken, and that is where the structural problem lives.

Languages differ in where they put the information that determines the shape of the target sentence. German subordinate clauses and Japanese main clauses put the verb at the end; English wants it early. A simultaneous system translating German into English that has heard “ich habe das Buch, das du mir empfohlen hast,” still does not know whether the speaker read it, bought it, lost it or hated it. It must either wait, or guess and revise.

The lag L is the number of source words the system consumes
before it emits its first target word.

  L small   low latency, more errors, more visible revision
  L large   fewer errors, higher latency

The trade-off is a property of the LANGUAGE PAIR, not of the
model:

  similar word order (Spanish -> Italian, Dutch -> German)
    small L works; the target can track the source closely

  verb-final -> verb-medial (German, Japanese, Korean, Turkish
  into English)
    small L forces the system to predict the verb before hearing
    it, and predicting wrong means either an ungrammatical
    sentence or a visible correction

Human simultaneous interpreters solve this the same way, with the
same costs: they lag by several seconds, they anticipate, and they
occasionally repair. That is not a limitation of the technology.

The consequence for design is worth stating plainly, because it is usually discovered late: a target of “under one second” for simultaneous translation is achievable for some language pairs and not for others, with the same system. Set latency expectations per pair.

Read-write policies

The policy is the rule deciding, at each moment, whether to consume more source (READ) or emit more target (WRITE). Three families:

• Fixed lag. The wait-k policy: read k source words, then alternate one write per read. Published by Ma and colleagues in 2019 as STACL. Trivial to implement, predictable latency, and blind to whether the current sentence actually needs the wait.
• Adaptive. The model decides per step whether it has enough context to commit. Better on average because it waits when the syntax requires waiting and not otherwise, at the cost of variable latency — which is harder to build a UI around.
• Re-translation. Retranslate the whole utterance from scratch on every update and replace the displayed text. Simple, uses an ordinary translation model with no simultaneous training, and gives the best final quality. Only viable for text output: you can rewrite a subtitle, and you cannot un-speak a sentence. The same stable-prefix technique used for ASR partials applies directly — freeze the prefix that has stopped changing and only revise the tail.

Which one you can use is decided by the output medium, not by preference. Subtitles can revise, so re-translation is usually right. Speech cannot, so a spoken system needs a policy that commits — and committing early is what produces the awkward mid-sentence repairs that make synthetic interpretation sound worse than it translates.

Measuring it

Two axes, and reporting one without the other is meaningless, because any system can be made better on one by getting worse on the other.

Add one thing neither captures, which matters more than either for a product: the revision rate. How often, and by how much, does displayed or spoken output get corrected? A system with excellent BLEU and constant visible revision is unusable, and no standard metric will tell you that.

Building one

1. Start consecutive. Wait for a turn, translate, speak. It is far simpler, it is correct, and for a two-party conversation with turn-taking it is frequently what people actually want. Do not build simultaneous until somebody has used consecutive and asked for it.
2. Segment on syntax, not on silence alone. A translation unit should be a clause. Using silence-based endpointing alone hands the translator fragments that cannot be translated, and the fragment boundary error is invisible in the ASR metrics.
3. Pass context across segments. Pronoun gender, formality, and terminology consistency all depend on what was said earlier. A translator called once per clause with no history produces a transcript that changes how it addresses somebody halfway through.
4. Pin your terminology. Product names, people’s names and technical vocabulary should be in a glossary the translator is given, not rediscovered per segment. This is the highest-value single intervention in a business context.
5. Show the source transcript when the medium allows. A bilingual reader can spot a mistranslation instantly if you show them both, and it costs nothing. This alone is a strong argument for the cascade in any interface with a screen.
6. Decide about voice preservation explicitly. Translating into a synthesised copy of the speaker’s voice is technically available and carries the whole consent question with it. A neutral voice per speaker role is usually the better product decision and always the simpler compliance one.