Voice AI Latency: How to Build a Streaming Pipeline That Answers in Under 500ms

A caller can't see a typing indicator, so silence is all they get. Every few hundred milliseconds of quiet after they stop talking reads as confusion, a dropped line, or a machine. Callers start repeating themselves, talking over the agent, or hanging up. Each of those costs you a booking, a qualified lead, or a support ticket that now needs a human.

Research on human conversation shows how small the target is. A study of ten languages (Stivers and colleagues, PNAS, 2009) found that the typical gap between turns sits around 200ms. A voice agent won't match that. Getting under roughly 500ms keeps it inside the range people accept without noticing. Above a second, most callers can tell something is off.

Where the time goes: the latency budget

From the caller's point of view, response latency is the sum of five terms:

Perceived latency = turn-detection wait + final transcript + LLM time to first sentence + TTS time to first audio + transport and playback buffering

Most teams tune the wrong term. They swap LLMs to save 100ms while their turn detector waits 700ms of silence before doing anything.

The table below is the worksheet we use to reason about a pipeline. The ranges are planning ranges for a well-built cascaded pipeline, not benchmarks. Your numbers depend on region, vendor load, prompt size and network, so measure your own at p50 and p95.

For TTS, ElevenLabs advertises around 75ms of model latency for its Flash models. That figure excludes network time, so treat it as a floor, not a total.

The main point of the table: in a sequential pipeline these terms add up. In a streamed pipeline they overlap. The caller hears the first sentence while the LLM is still writing the second. That overlap is the difference between a 2-second agent and a 450ms one.

Turn detection: the term most teams get wrong

Deepgram gives you two separate mechanisms. Tutorials often mix them up.

endpointing is voice-activity based. When Deepgram detects the configured number of milliseconds of silence, it marks the transcript speech_final: true. The default is 10ms, which is far too eager for conversation. Values of 200 to 500ms are a sensible starting range. Tune it against recordings of your real callers.

utterance_end_ms is word-timing based. Deepgram sends a separate UtteranceEnd message when it sees a gap after the last finalised word. The minimum value is 1,000ms, and it requires interim_results=true. It exists because background noise can stop the voice-activity detector from ever hearing silence. UtteranceEnd is your backstop for noisy lines, not your primary trigger.

Use both. Act on speech_final when it arrives. Treat UtteranceEnd as the safety net that stops a caller in a noisy car from waiting forever.

The trade-off is a business decision, not just a technical one. A shorter endpointing window makes the agent faster, but it interrupts people who pause mid-sentence. Callers reading out a policy number or an address pause a lot. If your calls are mostly structured data capture, a longer window costs less than repeated interruptions.

The LLM: time to first sentence, not total speed

What matters for voice is how fast the first complete sentence arrives. Total generation time matters far less. Three tiers are worth considering:

• ▸Small, fast hosted models (the Haiku tier of Claude, the mini tier of GPT, the Flash tier of Gemini). These are the right default for most voice agents. They are fast to first token and cheap per turn, and capable enough for scheduling, qualification and FAQ work.
• ▸Frontier hosted models. They are slower to first token and cost more. Use them only when the agent has to reason through something hard. Even then, consider a fast model for the spoken reply and a larger one behind a tool call.
• ▸Self-hosted open models (Qwen, Llama) on vLLM or SGLang. These are the choice when audio can't leave your infrastructure, which is common for Gulf healthcare and finance. Latency depends on your GPU, batch size and load more than on the model name. See our guide to on-prem inference speed.

Don't pick a model from a published table, including ours. Provider latency shifts week to week and differs by region. Independent trackers such as Artificial Analysis publish time-to-first-token measurements, and they're a reasonable shortlist. Then run 200 or more calls with your real prompt from your real region, and compare p95 rather than averages. The caller who hangs up is usually sitting at p95.

Two settings matter more than the model choice. Keep the system prompt short, because every token of it is processed before the first output token. And instruct the model to lead with a short sentence. "Sure, let me check that." reaches TTS far sooner than a 30-word opening.

TTS: stream text in, stream audio out

Use the ElevenLabs WebSocket input-streaming endpoint, not the REST endpoint. The REST call waits for the full text before you can start sending it. The WebSocket accepts text as the LLM produces it and returns audio chunks as they're generated.

Buffer LLM tokens until a sentence boundary, then send that sentence with flush so synthesis starts immediately. Request ulaw_8000 output when the caller is on a phone line. Twilio carries 8kHz μ-law, so matching it removes a transcoding step entirely.

The pipeline, in working code

The core rule is that sending text to TTS and forwarding audio to the caller must run concurrently. A common bug sends all the LLM text first and only then reads the audio back. That plays nothing until the full reply is synthesised, which throws away the benefit of streaming. The code below runs both directions at once with asyncio.gather.

It uses Twilio Media Streams for the phone leg and raw WebSockets for Deepgram and ElevenLabs, so you can see every message. Model names come from configuration because they change faster than your code should.

import asyncio, base64, json, os
import websockets
from fastapi import FastAPI, WebSocket
from openai import AsyncOpenAI  # any OpenAI-compatible endpoint, incl. vLLM/SGLang

app = FastAPI()
llm = AsyncOpenAI(base_url=os.environ.get("LLM_BASE_URL"), api_key=os.environ["LLM_API_KEY"])
LLM_MODEL = os.environ["LLM_MODEL"]
VOICE_ID = os.environ["ELEVENLABS_VOICE_ID"]

DG_URL = (
    "wss://api.deepgram.com/v1/listen"
    f"?model={os.environ['DEEPGRAM_MODEL']}"
    "&encoding=mulaw&sample_rate=8000&channels=1"   # Twilio's native format
    "&interim_results=true"                          # required for UtteranceEnd
    "&endpointing=300"                               # speech_final after 300ms silence
    "&utterance_end_ms=1000"                         # backstop; 1000ms is the minimum
    "&smart_format=true"
)
EL_URL = (
    f"wss://api.elevenlabs.io/v1/text-to-speech/{VOICE_ID}/stream-input"
    f"?model_id={os.environ['ELEVENLABS_MODEL']}"   # a Flash-tier model
    "&output_format=ulaw_8000"                       # no transcoding for Twilio
)
SENTENCE_END = (".", "!", "?", "؟")                 # include the Arabic question mark
SYSTEM_PROMPT = "You are a phone assistant. Answer in one or two short sentences."

class Call:
    def __init__(self, twilio: WebSocket):
        self.twilio = twilio
        self.stream_sid = None
        self.history = [{"role": "system", "content": SYSTEM_PROMPT}]
        self.finals: list[str] = []
        self.reply_task: asyncio.Task | None = None

    def speaking(self) -> bool:
        return self.reply_task is not None and not self.reply_task.done()

    async def send_audio(self, b64_audio: str):
        await self.twilio.send_text(json.dumps({
            "event": "media", "streamSid": self.stream_sid,
            "media": {"payload": b64_audio},
        }))

    async def stop_speaking(self):
        """Barge-in: cancel generation and flush audio Twilio has buffered."""
        if self.speaking():
            self.reply_task.cancel()
            await self.twilio.send_text(json.dumps(
                {"event": "clear", "streamSid": self.stream_sid}))

    async def end_turn(self):
        text = " ".join(self.finals).strip()
        self.finals.clear()
        if not text:
            return
        await self.stop_speaking()
        self.history.append({"role": "user", "content": text})
        self.reply_task = asyncio.create_task(self.reply())

    async def reply(self):
        sent: list[str] = []  # text sent to TTS (not necessarily heard)
        headers = {"xi-api-key": os.environ["ELEVENLABS_API_KEY"]}
        async with websockets.connect(EL_URL, additional_headers=headers) as tts:
            await tts.send(json.dumps({"text": " "}))  # open the stream

            async def llm_to_tts():
                buf = ""
                stream = await llm.chat.completions.create(
                    model=LLM_MODEL, messages=self.history,
                    stream=True, max_tokens=150, temperature=0.3)
                async for chunk in stream:
                    if not chunk.choices:
                        continue
                    buf += chunk.choices[0].delta.content or ""
                    if buf.rstrip().endswith(SENTENCE_END):
                        await tts.send(json.dumps({"text": buf + " ", "flush": True}))
                        sent.append(buf)
                        buf = ""
                if buf.strip():
                    await tts.send(json.dumps({"text": buf + " ", "flush": True}))
                    sent.append(buf)
                await tts.send(json.dumps({"text": ""}))  # end of input

            async def tts_to_caller():
                async for raw in tts:
                    msg = json.loads(raw)
                    if msg.get("audio"):
                        await self.send_audio(msg["audio"])  # already base64 μ-law
                    if msg.get("isFinal"):
                        break

            try:
                # Both directions at once: audio plays while the LLM is still writing.
                await asyncio.gather(llm_to_tts(), tts_to_caller())
            finally:
                if sent:
                    self.history.append({"role": "assistant", "content": " ".join(sent)})

    async def listen(self, dg):
        async for raw in dg:
            msg = json.loads(raw)
            kind = msg.get("type")
            if kind == "Results":
                text = msg["channel"]["alternatives"][0]["transcript"]
                if text and self.speaking():
                    await self.stop_speaking()      # caller is talking over us
                if msg.get("is_final") and text:
                    self.finals.append(text)
                if msg.get("speech_final"):
                    await self.end_turn()           # primary trigger
            elif kind == "UtteranceEnd":
                await self.end_turn()               # backstop for noisy lines

@app.websocket("/voice")
async def voice(ws: WebSocket):
    await ws.accept()
    call = Call(ws)
    dg_headers = {"Authorization": f"Token {os.environ['DEEPGRAM_API_KEY']}"}
    async with websockets.connect(DG_URL, additional_headers=dg_headers) as dg:
        listener = asyncio.create_task(call.listen(dg))
        try:
            async for raw in ws.iter_text():
                msg = json.loads(raw)
                if msg["event"] == "start":
                    call.stream_sid = msg["start"]["streamSid"]
                elif msg["event"] == "media":
                    await dg.send(base64.b64decode(msg["media"]["payload"]))
                elif msg["event"] == "stop":
                    break
        finally:
            await dg.send(json.dumps({"type": "CloseStream"}))
            listener.cancel()
            if call.speaking():
                call.reply_task.cancel()

The TwiML that connects an inbound call to this server:

<Response>
  <Connect>
    <Stream url="wss://your-server.example.com/voice" />
  </Connect>
</Response>

Three refinements to make before production:

• ▸Sentence splitting is naive. "Dr. Ahmed" and "3.5" both end in a period. Add an abbreviation list, and require a minimum length before flushing.
• ▸Flush the first clause early. On the first chunk only, flushing at a comma after a few words saves one clause of LLM time.
• ▸The history records what was sent, not what was heard. After a barge-in, the caller heard less than the log says. If the agent quotes prices or terms, track playback position so the record matches what was actually heard. The control is cutting the stored history at the last word the caller heard.

Barge-in: letting callers interrupt

People interrupt. An agent that keeps talking over them sounds broken, however fast it is.

Barge-in needs two things. You need to detect caller speech while the agent is talking, and you need to stop audio the caller hasn't heard yet. Cancelling the LLM and TTS tasks only stops new audio. Twilio has already buffered several hundred milliseconds of audio on its side. The clear message on the media stream discards that buffer, and without it the agent finishes its sentence after being interrupted. Stopping the audio is half the job; what the conversation history should record after the interruption is covered in the failure-modes post.

How you trigger barge-in is a tuning decision. Deepgram can emit SpeechStarted events (vad_events=true), which is fast but fires on coughs, TV audio and the agent's own echo on some handsets. Triggering on an interim transcript that contains real words, as the code above does, is slower but makes far fewer false stops. For a noisy caller base, require two or more words before cutting the agent off.

Transport: phone lines versus WebRTC

On a phone call, the carrier leg decides your transport. Twilio delivers 8kHz μ-law over a WebSocket, and the job is to match that format end to end and host close to the carrier's media region.

Browser and in-app voice is different, and the transport choice matters there. WebSockets run over TCP, and TCP delivers every packet in order. One lost packet stalls everything behind it until it's retransmitted, which is called head-of-line blocking. For a file download that's correct. For live audio, it means stutters and latency spikes on any lossy mobile connection.

WebRTC is the right transport for browser and app clients. It runs media over UDP with the Opus codec. When a packet is lost, the codec conceals the gap and the stream continues. It also includes jitter buffering and echo cancellation, which you'd otherwise build yourself.

At volume, put a media server between clients and your AI workers. A Selective Forwarding Unit (SFU) such as LiveKit or Janus terminates the WebRTC connections and forwards audio to your worker pool. That separates connection handling, which is network-heavy, from AI orchestration, which waits on APIs. The two then scale independently. LiveKit also ships an agents framework built around this pattern. That's worth evaluating before you build the plumbing yourself.

Scaling past a handful of calls

A hosted-model pipeline spends most of its time waiting on network calls, not computing. Sessions per server are therefore limited by event-loop health, memory per connection and outbound socket limits, not CPU cores. Nobody can give you a sessions-per-server figure that holds for your stack. Load-test it.

The sizing formula is simple:

Instances needed = peak concurrent calls ÷ sessions per instance (measured at your p95 latency target) × headroom factor (1.3 to 1.5 is a sensible planning range)

Measure sessions per instance by ramping synthetic calls until p95 response latency crosses your target, not until the server falls over. Latency degrades well before anything crashes.

Three walls usually arrive before compute does:

1. ▸Vendor concurrency caps. STT and TTS providers limit concurrent streams by plan tier. Find out your limits before launch day, not during a campaign.
2. ▸Blocking code in the event loop. One synchronous database call or CPU-heavy step inside the async loop adds latency to every call on that instance.
3. ▸Sticky state. Keep per-call state inside the connection handler, as above, so any instance can take any call. If a call has to move between instances, put shared state in Redis.

Self-hosted LLMs change the picture. The GPU becomes the bottleneck, and per-call latency rises with concurrent load as batches grow. Size that tier from a load test of your model on your hardware, at your target p95.

Arabic and code-switching

The pipeline is the same for Arabic. The settings change.

Pin the STT language when you know it. Before relying on Deepgram's multilingual mode for mid-sentence Arabic-English switching, check that the mode covers your language pair. Gulf callers switch languages constantly, and dialect performance varies by vendor. Testing on Khaleeji, Egyptian or Levantine audio that matches your callers is essential. A one-week dialect test for the STT engine settles it on your own recordings.

For TTS, use a voice built for Arabic rather than a multilingual voice that happens to read it. Add ؟ to your sentence-boundary list, or Arabic questions will never trigger an early flush. Tell the LLM to reply in the caller's language and to keep replies short, since Arabic replies often run longer than English.

Frequently asked questions

Should we use a speech-to-speech model instead of this cascaded pipeline? Native speech-to-speech models can cut latency and sound more natural, but the cascaded pipeline is still the safer choice if you need compliance checks, exact pronunciation control, or a specific Arabic dialect. See how to choose between speech-to-speech and a cascade by call type.

Why does our agent feel slow when every vendor dashboard shows fast numbers? Vendor dashboards measure their own stage, not the gap the caller hears. Usually the turn-detection wait or a sequential step is the problem. Log a timestamp at each stage boundary for every turn: speech_final, first LLM token, first full sentence, first TTS audio, first audio sent to the carrier. The largest gap is your fix.

Can a pilot that works for one tester handle a real call campaign? Often not without changes. Single-user pilots hide vendor concurrency caps, blocking code and missing barge-in handling. A pilot passes when it holds your p95 latency target at your expected peak concurrent calls in a load test. Make that the acceptance criterion before anyone schedules a launch.

Before changing models, add the five timestamps above to your pipeline and run 50 real calls. One term will dominate, and it's usually the turn-detection wait.