Voice agents: what breaks on a real phone call
A voice agent is three ordinary pieces glued to a phone line. The glue is where it fails, and a polished demo is designed to hide exactly that.
Someone calls your business at 7:40 on a Tuesday night. They want to move an appointment, they are in a car with the window down, and halfway through the greeting they cut in with "mira, es que". Whether your voice agent survives that moment has almost nothing to do with how smart the model is.
Most voice AI you see sold is demoed on a laptop, in a quiet room, by the person who built it, saying sentences it has heard before. That demo proves the pipeline is wired. It proves nothing about the call above.
What a voice agent actually is
Three ordinary components in a line. Speech to text turns audio into words. A language model, usually with a fixed system prompt and a few functions it is allowed to call, decides what to say and what to do. Text to speech turns that back into audio.
Around those three sits the part nobody demos: telephony. A carrier leg, a media stream, echo handling, and turn-taking logic that decides when the caller has stopped talking and whether an interruption should kill the current sentence. The intelligence is rented from an API. The engineering is in the plumbing, and the plumbing is what your callers experience.
Where they fail
Failure in voice is not usually a wrong answer. It is the agent talking over the caller, or leaving a two-second hole that makes the person say "hello?", or cheerfully confirming a name it never heard correctly. Recognition errors compound: the model answers confidently about whatever the transcript said, not about what the caller said.
- Proper nouns over the phone. Dominican surnames, colmado names, street names in Los Alcarrizos. These are the words a general transcriber is worst at, and they are the words that matter most in a booking.
- Long digit strings. Cédulas, order numbers, phone numbers read back in groups, the difference between "dieciséis" and "sesenta".
- Code-switching mid-sentence, which is normal here and which some transcription setups handle badly.
- Noise: moto engines, a colmado speaker, a call on speakerphone in a car. Accuracy drops hard and the agent does not know it dropped.
- Silence and hesitation. A caller thinking for three seconds is not a caller who finished their turn.
- Anything requiring judgement about money, a complaint, or a person who is already angry. The correct behaviour is to stop and hand off.
Latency is the other axis. Round trip from the caller's last syllable to the first syllable of the reply needs to sit well under a second to feel like a conversation. Every added step costs you part of that budget, which is why an agent that looks up live inventory before answering often feels worse than one that promises a callback.
Assume the agent will be wrong on a call that matters. Design the handoff first and the script second.
Telling a real implementation from a demo
Ask to call the number yourself, from your own mobile, outdoors. Not a web widget, not a recording. Then interrupt it twice, give it a name it cannot guess, and go quiet for four seconds in the middle of a sentence. You will learn more in ninety seconds than from any slide.
- Ask what happens when the model API is slow or down. A real build has a timeout and a fallback, usually a transfer or a message taken.
- Ask where the transcript and the recording land, and who can read them. If the answer is only "the dashboard", it is not integrated with anything.
- Ask for the escalation path in specifics: which number, which hours, what happens outside them.
- Ask what the agent is not allowed to do. A build with no refusal list has not been thought through.
- Ask how the calendar or CRM write actually happens, and what happens if that write fails after the agent already said "listo".
A voice agent earns its place on the calls that are repetitive, short, and low-stakes: hours, directions, confirming and moving appointments, taking a message with the details already structured. Point it at your hardest calls and it will embarrass you in public, one caller at a time. Point it at the twenty identical calls a day nobody wants to answer and it quietly pays for itself.