Building a Phone Agent: SIP, WebRTC and the Audio Path

A phone agent is judged on one number: the gap between the caller finishing a sentence and hearing a reply. That gap is a sum of about a dozen hops, most of which are invisible from your application code. This page names every one of them and adds them up.

The path, hop by hop

Start with the physical route, because the software terms sit inside it. A caller on the public telephone network reaches you through a SIP trunk from a carrier or a CPaaS provider; a caller in a browser reaches you through WebRTC. Both end at a media server that hands you RTP packets containing 20 milliseconds of audio each.

INBOUND  (caller speaks -> your endpointer notices they stopped)

  h1  capture + packetisation   handset mic, codec framing.  One RTP
                                packet carries one frame; 20 ms is the
                                near-universal telephony choice, so the
                                encoder always holds a full frame before
                                it can send anything.  Opus adds a few ms
                                of algorithmic look-ahead on top; G.711
                                adds none.
  h2  access network            mobile radio or fixed broadband, one way.
                                The most variable term in the whole path
                                and the one you cannot touch.
  h3  carrier transit           PSTN / interconnect, plus transcoding at
                                any gateway that changes codec.
  h4  provider media path       your CPaaS or SBC to your media server.
  h5  jitter buffer             de-jittering at your end.  Adaptive, and
                                the largest network term you actually
                                control.
  h6  frame handoff             media server -> your process -> VAD.

  --- the caller has now stopped speaking, and you know it ---

  e   endpoint wait             min_silence.  A policy, not a system.
  r   ASR finalisation          from endpoint to final transcript.
  m   model time to first token
  s   TTS time to first audio byte

OUTBOUND (your first audio byte -> the caller hears it)

  h7  packetisation             your encoder frames the audio: another
                                20 ms held before the first packet leaves.
  h8  provider media path
  h9  carrier transit
  h10 access network
  h11 caller jitter buffer + decode + playout

Two structural facts fall out of that list before any number is attached. Packetisation is paid twice, once in each direction, because neither encoder can emit a packet it has not finished filling. And the jitter buffer is paid twice as well — once at your end, once at theirs — and neither is a fixed quantity, because adaptive buffers grow when the network gets worse.

The budget as an equation

Write the response gap as the caller experiences it: from their last syllable leaving their mouth to the first syllable of the reply arriving at their ear.

gap  =  D_in  +  e  +  r  +  m  +  s  +  D_out

  D_in   = h1 + h2 + h3 + h4 + h5 + h6      inbound one-way delay
  D_out  = h7 + h8 + h9 + h10 + h11         outbound one-way delay

  e      endpoint wait (min_silence)
  r      ASR finalisation after the endpoint fires
  m      model time to first token
  s      TTS time to first audio byte

Every term is in milliseconds. Fill each one from your own
measurements; nothing on this page supplies a value for you.

The two useful reference points to compare the result against:

  ITU-T G.114 recommends a one-way mouth-to-ear delay at or below
  150 ms for general conversational quality, treats 150-400 ms as
  acceptable where the parties are aware of the transmission, and
  regards anything above 400 ms as unacceptable for interactive
  use.  Note that G.114 is about D_in alone -- your gap contains
  D_in AND D_out AND four processing terms.

  Work on human turn-taking (Stivers et al., PNAS 2009, ten
  languages) finds response offsets clustering around a couple of
  hundred milliseconds.  That is the standard a listener applies
  without being asked to.

So the honest framing: a carrier-grade telephone path spends a
substantial part of the human turn-taking budget before your
software has done anything at all.  Your four terms are competing
for what is left.

This is why “we made the model faster” so often fails to change how the product feels. If m is a fifth of the sum, halving it moves the gap by a tenth. Measure the whole path before optimising any part of it.

One illustrative fill-in

ILLUSTRATIVE ONLY -- substitute your own measurements.

INBOUND
  h1  packetisation                20 ms
  h2  access network               25 ms
  h3  carrier transit              20 ms
  h4  provider media path          10 ms
  h5  your jitter buffer           40 ms
  h6  frame handoff                 5 ms
                                  ------
  D_in                            120 ms

PROCESSING
  e   endpoint wait               600 ms
  r   ASR finalisation            150 ms
  m   model time to first token   400 ms
  s   TTS time to first audio     200 ms
                                 ------
                                 1350 ms

OUTBOUND
  h7  packetisation                20 ms
  h8  provider media path          10 ms
  h9  carrier transit              20 ms
  h10 access network               25 ms
  h11 caller jitter + playout      40 ms
                                  ------
  D_out                           115 ms

  gap = 120 + 1350 + 115        = 1585 ms

Read what that says rather than the number itself:

  network + telephony            235 ms   (15%)  mostly not yours
  endpoint wait                  600 ms   (38%)  a policy you chose
  ASR + model + TTS              750 ms   (47%)  four sequential services

The largest single line is a parameter someone typed, and the
second largest is three services run one after another that do not
have to be.

What overlaps and what does not

Three of the four processing terms can be partly hidden behind each other. Doing so is the difference between a demo and a product.

1. Overlap e with nothing — shorten it instead. The endpoint wait is pure dead time by construction: it is time spent confirming that nothing is happening. It cannot overlap anything. The only lever is making the decision smarter, which is what semantic endpointing is for: shorten e when the partial transcript looks complete, lengthen it when it ends in a conjunction or an unfinished number.
2. Overlap r with m. Issue the model request against the latest partial transcript the moment the endpoint fires, rather than waiting for the ASR final. When the final arrives, compare: if it matches what you sent, you have saved the whole of r; if it differs materially, cancel and reissue. The cost is an occasional wasted call.
3. Overlap m with s. Stream the model’s tokens into the TTS as they arrive and start synthesising at the first sentence boundary rather than at the end of the reply. This converts m from “time to complete the answer” into “time to first token” plus the time to reach a clause boundary. It is usually the single largest software win available, and it is the reason time to first token matters far more than tokens per second for a voice product.
4. Overlap s with h7 by streaming audio out. Push the first audio frames to the media server as the TTS produces them, not after the file is complete. A TTS that only returns whole utterances makes this impossible and should be disqualified on that basis alone.

Sequential:   gap = D_in + e + r + m_total + s_total + D_out

With all three overlaps:

  gap ~= D_in + e + max(r, m_first_token) + s_first_audio + D_out

where m_first_token is time to FIRST token, not the whole reply,
and s_first_audio is time to the first audio byte of the first
clause.  The rest of the reply is generated and synthesised while
the caller is already listening to its beginning -- which works
only if generation is faster than speech, roughly 150 words per
minute.  If it is not, the caller hears gaps mid-sentence, which
is worse than one gap at the start.

That last constraint is the one people discover in production. Check it explicitly: if your model and TTS together cannot sustain output faster than the voice speaks it, streaming mid-utterance produces stuttering rather than speed, and you are better off buffering the first sentence.

The codec decides your accuracy ceiling

The audio path does not only cost you milliseconds. It costs you bandwidth in the acoustic sense, and that puts a ceiling on recognition accuracy no ASR model can lift.

Two consequences worth designing around. First, upsampling 8 kHz telephony audio to the 16 kHz your ASR model expects is a format conversion, not a restoration: the missing band is gone and no resampler invents it. Second, every transcode in the path is a generation of lossy compression, and a call that goes through two of them is measurably harder to transcribe. Ask your provider for the codec on the leg you actually receive, not the one their marketing page describes.

This is also the largest single reason phone-call word error rates are worse than the figures quoted for clean audio, and why an accuracy number measured on studio recordings tells you nothing about your call centre. Measure on your own audio, with your own normaliser.

Instrumenting it

You cannot fill in the budget from application logs alone, because half the terms happen before your code sees anything. Three measurements between them cover the whole path.

1. Software terms, from one monotonic clock. Stamp six events on every turn, all from the same clock in the same process: last speech frame observed, endpoint fired, ASR final received, model request sent, first model token received, first TTS audio byte emitted. The differences give you e, r, m and s exactly, with no estimation.
2. Network terms, from RTP. RTP carries a timestamp and a sequence number per packet, and RTCP receiver reports carry jitter and round-trip time. Your media server exposes these; log the round trip and the current jitter buffer depth per call. Half the round trip is a serviceable estimate for h2 + h3 + h4 in each direction, and the buffer depth is h5 directly.
3. The true end-to-end number, once, physically. Place a real call from a real handset on the network your callers actually use. Record the handset’s own audio — both what you say into it and what comes out of it — into one stereo file. The gap between the end of your utterance on one channel and the start of the reply on the other is the real number, including every term you cannot instrument. Do this on a mobile network and on a landline; they will not agree.

Report all of these at p50 and p95, never as a mean. A single cold-start or a single jitter-buffer expansion drags a mean past every call anybody actually experienced, and on a phone system the tail calls are the ones that generate complaints.

The build order

Build it in the order that lets you measure at each stage, because a pipeline assembled all at once cannot be debugged.

1. Answer and echo. Accept an inbound call and play back the caller’s own audio with a fixed delay. This validates the media path, the codec, the packetisation and your buffering, and gives you D_in + D_out as a single measurable number before any AI exists.
2. Add VAD and log only. No responses. Log every speech and silence boundary, then build the pause histogram described in endpointing and choose e from your own data.
3. Add streaming ASR. Print partials and finals to a console beside the audio. Confirm that finals arrive when you expect and that the transcript is usable before anything reads it.
4. Add TTS with a fixed script. No model yet. Measure s, confirm streaming works end to end, and confirm you can stop playback mid-utterance — which is the prerequisite for barge-in.
5. Add the model last. By this point every other term is measured, so when the gap changes you know exactly which number moved.
6. Then add barge-in, and expect it to break things. Interruption interacts with every component before it, which is why it goes last and gets its own page.

One operational warning that is not about latency at all: inbound and outbound calling are regulated differently almost everywhere, automated outbound calling especially so, and call recording has its own rules again. Read recording, consent and retention before the first outbound call, not after.