Voice agents in production: what breaks first

Text-based agents get a second chance. A user can reread a bad answer, refresh the page, correct the prompt. Voice agents get zero seconds. The moment the response is wrong, awkward, or slow, the caller has already formed an opinion about your brand, and it is not the opinion the marketing team wanted.

Voice also fails in more ways than text. Speech recognition can mishear a name. Text-to-speech can mangle an address. Network jitter can insert a two-second pause that reads as either confusion or dishonesty depending on what came before. Each of these failure modes is independent, and each of them compounds.

Everything that follows is what we have learned shipping voice into medical intake, insurance renewals, and outbound qualification. The lessons are ordered by how quickly they break in production, so read them in order and design defences in the same order.

Users forgive an agent that gets one thing wrong. They do not forgive one that pauses for two full seconds before speaking. Every extra three hundred milliseconds of latency reduces perceived intelligence by more than raw accuracy improvements ever add. This is uncomfortable for engineers who spent a decade optimising for correctness, but the data on live calls is unambiguous.

Design for a hard latency ceiling of eight hundred milliseconds from end of user speech to start of agent response. Anything above one point two seconds and users will start talking over the agent, and the conversation collapses. Once the caller starts to interrupt, recovery is almost impossible without a human handoff.

The main levers are streaming automatic speech recognition, a smaller reasoning model for the first turn with a larger one for escalations, and pre-warmed text-to-speech. Skip any vendor that cannot demonstrate all three in a live call, on a real network, with a real script that includes interruptions.

One trick that pays off surprisingly well: emit a short filler acknowledgement within two hundred milliseconds while the reasoning model is still working. A well-placed one moment. or checking that. buys you an extra half second of latency headroom without the caller noticing. Overuse it and the agent sounds evasive, so cap it at once every three turns.

A voice agent that cannot cleanly hand off to a human is a liability. Design the escalation logic before the happy path, because the escalation logic is what determines whether a bad conversation ends politely or ends on Twitter.

Escalation triggers should include: three consecutive low-confidence turns, any mention of a competitor or complaint keyword, an explicit request for a manager, any topic outside the trained scope, and any detected emotional distress signal above threshold. Each of these routes to a different human queue with the transcript attached, because a billing complaint needs different context than a technical outage.

The transcript matters more than the routing. A rep who picks up an escalation without the context of what the agent said before will make things worse. Include the last three turns verbatim, the agent's confidence scores, and a one-line summary of what the caller wants. If your human queue does not have a place to display this, build one before you launch.

The other side of escalation is graceful failure. When no human is available, the agent needs a scripted apology that captures a callback number, sets a specific expectation for response time, and creates a ticket in your system automatically. Never leave a caller on hold for a queue that does not exist.

The call itself is not the product. The CRM write after the call is. Every voice agent needs a structured extraction step that turns the transcript into fields your operators actually use: intent, next action, sentiment, commitments the agent made, and any dates or dollar amounts mentioned.

This is also where most agents leak value. Skipping the extraction step means the operator still has to relisten and take notes, which erases half the ROI and creates a shadow process where nobody trusts the CRM data. If the CRM write is unreliable, the operators will start listening to every call by default, and you have paid for an expensive answering machine.

Bake evaluations for the extraction step separately from the conversation quality. Different failure mode, different fix. Conversation evals measure whether the caller had a good experience. Extraction evals measure whether the CRM ended up with the right data. A call can be excellent on the first axis and worthless on the second.

Our benchmark is ninety percent field-level accuracy on the top five fields, measured on a golden dataset that includes messy real-world calls, not just clean pilot examples. Anything below that and the operators lose trust in the automation and revert to manual review.

Recording consent, PII redaction, and jurisdictional call rules vary by state and country. Build the compliance layer as configuration, not as branching prompts, because prompts drift and configuration is auditable.

For regulated industries, we also add a supervisor model that reviews every call transcript for policy violations within an hour of the call ending. This is cheap insurance and it catches the one in five hundred conversation that would have become a regulator letter. The supervisor does not need to be as capable as the primary agent; it just needs to flag risk categories reliably.

One piece of specific advice: never let the agent quote a price, a delivery date, or a legal commitment that is not backed by a verified data source. If the CRM says the customer is on plan A, the agent can confirm plan A features. If the agent is inferring the plan from context, the answer should be a soft handoff to a human. The extra seconds cost less than a mispriced contract.

Log everything. Retention windows will change over the life of the system, and the only way to shorten them later without losing continuity is to have logged the raw data with clear timestamps and consent flags from day one.

Latency under eight hundred milliseconds measured across at least two hundred test calls, not one. The distribution matters more than the average, because the calls above the ninetieth percentile are the ones that damage the brand.

Escalation triggers tested with adversarial scripts, not just happy path. We keep a library of one hundred adversarial scripts covering irate callers, non-native speakers, background noise, and out-of-scope questions. Every launch runs the full library.

CRM writes verified against a golden dataset with over ninety percent field-level accuracy on the top five fields, measured on messy real-world calls.

Cost per call ceiling wired into runtime, with automatic failover to a cheaper path when volume spikes. A voice agent with no cost ceiling is a bill waiting to happen.

A named on-call engineer for the first four weeks post-launch. This is the single highest-leverage line in the whole launch plan. Most catastrophic voice launches were technically fine on day one and quietly broken by day nine because nobody was watching.

A weekly review of ten random calls with a human operator in the loop, for at least the first quarter. The evaluation harness catches regressions in aggregate; humans catch the qualitative drift that no eval will flag.

If you get all seven right, you are ahead of ninety percent of production voice deployments we have audited. If you skip any of them, expect to be back in this checklist by month three.