Voice AI Failure Modes in Production: Latency, Barge-In, Dead Air and Code-Switching

A voice agent tested in a quiet office, by people who wait politely for their turn, will pass every test. Then a real caller phones from a car. They interrupt halfway through a sentence and drop English words into Arabic. The same system now talks over them, goes silent, or confirms things it never said.

None of these failures is random. They are predictable, they recur wherever a voice agent meets real callers, and each one has a known control. Each one also has a price if nobody engineers for it.

Failure 1: Latency that compounds across the pipeline

People expect a reply almost instantly. A cross-language study of turn-taking by Stivers and colleagues (PNAS, 2009) found the typical gap between speakers sits around 200 milliseconds. A voice agent will not match that. But every extra half second gets noticed. A later paper in Language, Cognition and Neuroscience, Conversational expectations get revised as response latencies unfold appeared in 2019. It tracked how a listener's reading of a reply shifts while the silence runs.

The business consequence is specific. If the agent is slow, callers assume it did not hear them and repeat themselves. The repeat collides with the agent's late reply. Now you have two people talking at once, and one of them is software.

The cause is usually a sequential pipeline. The system waits for the caller to stop, then transcribes. It then asks the language model for a complete answer and converts the whole answer to speech. Each stage waits for the one before it. As an illustrative range, those stages add up to between 1.5 and 2.5 seconds of silence before the caller hears anything.

The fix is to overlap the stages. Audio streams into speech recognition while the caller is still talking. The model's reply streams out word by word. The first complete phrase, such as "I can help with that," goes to the voice engine while the rest is still being written. As a planning range, a well-built streaming pipeline lands between 600 and 900 milliseconds. The latency budget and working streaming code are in the companion post. Getting under 500 milliseconds takes either careful tuning end to end or a native speech-to-speech model, and each has its own trade-offs by call type.

Failure 2: Barge-in that breaks the conversation's memory

Barge-in means the caller interrupts the agent. A basic build ignores it. The agent plays its full, long answer while the caller says "no, no, that's wrong" underneath. Average handle time goes up, and so does the share of callers asking for a human.

Production systems listen for speech while the agent talks. When the caller starts speaking, playback stops at once. On Twilio Media Streams, for example, the application sends a clear message to flush audio already buffered for the caller. Twilio's Media Streams WebSocket messages documentation defines both clear and the mark events used below. Its Gather verb has a separate bargeIn setting for Twilio's own text-to-speech, which is a different mechanism.

Stopping the audio is the easy half. The hard half is memory.

Suppose the model wrote "Your appointment is confirmed for Tuesday at 3 PM". The caller cut in after "Your appointment is". By default the conversation history still holds the full sentence. The model now believes the caller heard the day and time. Two turns later it says "as I mentioned, Tuesday works," and the caller has no idea what it means.

The control is to cut the stored history at the exact point the caller stopped hearing. That needs two things. The voice engine must report timing for each word, which many engines provide. The telephony layer must confirm what actually played; Twilio's mark events do this. The illustrative trace below shows the result.

# Illustrative trace, not from a client system
t=0.00  agent_planned: "Your appointment is confirmed for Tuesday at 3 PM."
t=0.00  playback_start
t=1.10  mark: "Your appointment is"
t=1.18  caller_speech_detected (transcript: "wait, which clinic")
t=1.19  playback_clear sent
t=1.20  history_write: agent said "Your appointment is" [interrupted]
        (unheard: "confirmed for Tuesday at 3 PM.")
t=1.21  next turn: model sees caller never heard day or time

Get this wrong and the agent acts on facts the caller never heard. Get it right and the agent repeats the date when it matters.

The false barge-in, including the agent interrupting itself

The other side of barge-in is interruptions that are not real. A cough, a siren, or a door slam can stop the agent mid-sentence. The agent then answers the noise, apologises, and loses its thread. Do that three times and the caller hangs up.

The worst case is the agent interrupting itself. On speakerphone or in a car, the agent's own voice leaks back into the caller's microphone. A system without echo handling hears that leak as speech and cuts itself off, sometimes in a loop.

The controls, in the order we apply them:

• ▸Echo cancellation on the inbound audio, so the agent's own voice is removed before speech detection sees it.
• ▸A stricter threshold while the agent is speaking, since that is exactly when echo and false triggers are most likely.
• ▸Words, not just sound. Require the transcript to contain real words, held for a minimum duration, before treating it as an interruption. A cough produces energy but no words.
• ▸Tuning per environment. A line for a construction firm hears different background noise than a private clinic. Set thresholds from recordings of that line, not defaults.

Failure 3: Dead air while the system does real work

Voice agents exist to do things: look up an order, check a slot, book an appointment. That means calling your other systems, and those systems are not always fast.

Say the booking system takes four seconds to answer. The line goes silent. After two or three seconds the caller says "Hello? Are you there?" That new speech starts a fresh turn while the first request is still running. The agent now has two jobs in flight and one conversation to manage. It often loses its place.

The more expensive version happens when the caller hangs up mid-request. The booking may go through anyway. Or the caller rings back, books again, and you have a double booking or a double charge that someone must clean up by hand.

Four controls contain this:

• ▸Filler speech in parallel. The moment the agent starts a lookup, it says "Let me pull that up for you" while the request runs. The caller hears progress, not silence.
• ▸A hard timeout with a scripted fallback. If the lookup takes longer than a set limit, the agent says so plainly and either keeps waiting or offers a callback. It never guesses. The voice engine needs the same treatment: TTS timeouts, concurrency caps and a second voice engine are covered in their own post.
• ▸Results come only from the system. The model reads the actual response from the booking system. If that system returns an error or nothing, the conversation takes a defined error path. The model is never allowed to infer that a booking succeeded.
• ▸Duplicate protection. Each action carries a unique request key, so a retry or a second call cannot create a second booking.

We build this using stateful orchestration, where each action is a separate, logged step with its own success and failure paths.

Failure 4: Code-switching in Gulf calls

Gulf callers rarely stay in one language. A Khaleeji Arabic sentence will carry an English product name, an English number, or a phrase like "cancel my appointment". The agent has to handle all of it without stalling.

There are two separate failures here.

The transcript breaks. Lock speech recognition to Arabic and the English phrase gets forced into Arabic sounds, producing nonsense text. The model then answers the nonsense. Leave the language on automatic detection and some engines need a second or more of audio to decide. That adds delay at the start of every turn. Engines differ widely on mixed Arabic and English, and on dialect. Test candidates on recordings from your own lines before committing; the one-week test for Arabic speech-to-text engines sets out how.

The reply switches language. A common model failure is answering in English because the caller used one English noun. To the caller, the agent has suddenly stopped speaking their language. The control is a firm rule in the agent's instructions and routing: reply in the caller's main language unless they clearly ask to switch. The voice must follow the same rule, including how it reads numbers, dates, and times aloud. Which Arabic voice reads dialect text convincingly is settled by a listening test on your own script, not by a vendor page.

Failure-mode reference

What a minute of AI handling costs

Do not trust anyone's per-minute claim, ours included, without running your own numbers. Your cost per AI-handled minute is the sum of speech recognition, model usage, voice generation, and telephony for that minute. Compare it with your fully loaded cost per human-handled minute. Then multiply the difference by the minutes the agent fully resolves, not the minutes it touches. Calls that escalate after three minutes of AI cost you both. A per-minute cost model with containment rate works this through line by line.

Frequently Asked Questions

Q: Why does our voice prototype work in testing but fail when customers call from their cars? A quiet room hides speech-detection problems. Road noise, wind, and Bluetooth microphones trigger false interruptions and fragment the audio sent for transcription. Speakerphone also leaks the agent's own voice back into the line. You need echo cancellation and thresholds tuned on real phone audio from that line.

Q: Can we use a native speech-to-speech model on a customer service line? For open conversation, often yes, and the latency is excellent. For flows that must query your systems, verify identity, and follow a compliance script, a streaming pipeline gives you more control. That pipeline uses a text model from a family such as Claude, GPT, or Qwen for reasoning. You can inspect and log each step, and enforce rules between them.

Q: How do we stop the agent from confirming a booking that never happened? Separate the conversation from the action. Each booking is a distinct step whose result comes only from your booking system. If that system errors or times out, the call follows a scripted error path. The model is never allowed to guess the outcome. For a clinic line, the slot hold and the fixed booking sequence make this concrete.

Before you put a voice agent in front of customers, take the table above and test each row on a real call. The rows that fail are your engineering plan, and they are far cheaper to fix before launch than after.