We’ve seen a lot of voice AI demos over the last year that sound impressive in isolation and then fall apart the moment real callers show up.
This piece is an attempt to write down what actually breaks in production voice systems, and why most of those failures are not model-related. Latency, turn-taking, barge-in, state handling, and escalation end up mattering more than prompt quality or model choice once you put traffic on a phone line.
The core argument is that voice AI should be designed as a real-time system with strict constraints, not as a sequence of API calls glued together. We also go into why many teams underestimate latency by measuring it in the wrong place, and how architecture choices quietly define what is even possible conversationally.
This is written from the perspective of building and running these systems, not from a research angle. No claims about “human-like” agents. Mostly lessons learned the hard way.
We’ve seen a lot of voice AI demos over the last year that sound impressive in isolation and then fall apart the moment real callers show up.
This piece is an attempt to write down what actually breaks in production voice systems, and why most of those failures are not model-related. Latency, turn-taking, barge-in, state handling, and escalation end up mattering more than prompt quality or model choice once you put traffic on a phone line.
The core argument is that voice AI should be designed as a real-time system with strict constraints, not as a sequence of API calls glued together. We also go into why many teams underestimate latency by measuring it in the wrong place, and how architecture choices quietly define what is even possible conversationally.
This is written from the perspective of building and running these systems, not from a research angle. No claims about “human-like” agents. Mostly lessons learned the hard way.