Where the second goes in an AI phone call

People forgive a voice agent a great deal. They forgive a misheard word, a clumsy sentence, even a wrong answer if it is corrected. What they do not forgive is the pause. A gap of much more than a second reads as broken, and callers respond by repeating themselves, which arrives as an interruption, which makes the next gap worse.

So latency is worth taking apart properly. In a live call the delay between the caller finishing and the agent starting is made of five things.

The five costs

1. 01Endpointing. Deciding the caller has actually stopped rather than paused for breath. This is a judgement call, not a measurement, and it is usually the largest single item.
2. 02Transcription. Turning the final audio into text. Streaming recognisers have most of the utterance transcribed before the caller stops, so the marginal cost here is small.
3. 03Generation. The model producing a reply, of which only the time to the first token matters.
4. 04Synthesis. Turning text into audio, again only as far as the first chunk.
5. 05Transport. Getting the audio back down the line, which is small and mostly out of your hands.

The instructive thing about that list is that items two through four do not have to happen in sequence, and item one is a tuning problem rather than an engineering one.

Endpointing is a tradeoff, not a bug

Voice activity detection has to wait some amount of silence before declaring the turn over. Set that window short and the agent interrupts people mid-sentence, particularly anyone who pauses to think, reads a number off a screen, or speaks a language whose rhythm the tuning did not anticipate. Set it long and every exchange feels sluggish.

There is no setting that is correct for all callers, which means this is not a number to optimise once. It is a decision about which failure you would rather have, and the right answer differs by use case. An agent taking a card number should wait longer. An agent doing rapid qualification should cut in sooner.

Streaming is where the real wins are

The naive pipeline waits for each stage to finish before starting the next: full transcript, then full reply, then full audio. That serialises costs which have no reason to be serial, and it makes the total roughly the sum of the parts.

The fix is to start each stage on partial output from the one before. Transcription runs on the audio as it arrives. Generation starts on the transcript the moment the turn ends. Synthesis begins on the first clause of the reply rather than on the finished sentence, and playback begins on the first chunk of audio rather than on the finished file. Done properly, the caller hears the beginning of the answer while the model is still deciding how it ends.

The catch is that a partially synthesised reply is a commitment. Once the first clause is playing, the agent cannot change its mind about the sentence, so anything that might invalidate the reply has to resolve before synthesis starts rather than during it.

Two details that cost more than they should

Markdown is the first. Models reliably produce asterisks, bullets and numbered lists, because that is what they were trained to produce, and a synthesiser will happily read the punctuation aloud or trip over it. Stripping formatting before synthesis is a small piece of code that removes a whole class of strange-sounding calls.

The second is the temptation to fix latency by asking for shorter replies. It works, and it makes the agent worse. Instructing a model to be brief flattens it: answers lose the qualification that made them useful, and callers ask follow-up questions to recover the detail, which costs more total time than the long reply would have. If the first chunk of audio arrives quickly, reply length barely matters to how fast the agent feels.

Perceived latency is time to first audio, not time to last.

Measure the gap, not the components

Component timings are useful for finding a regression and misleading as a target, because they do not add up to the thing the caller experiences. The number that matters is the wall clock between the caller’s last word and the first audible syllable of the reply, measured on real calls rather than on a synthetic loop, and looked at as a distribution.

The tail is where the damage is. A median that looks excellent alongside a long tail produces an agent most people find quick and some people find broken, and the ones who find it broken are disproportionately the ones on a bad connection, speaking a second language, or calling about something complicated. Those are rarely the callers you can afford to lose.