The baseline configurations all note <2s and <3s times. I haven't tried any voice AI stuff yet but a 3s latency waiting on a reply seems rage inducing if you're actually trying to accomplish something.
I've generally observed latency of 500ms to 1s with modern LLM-based voice agents making real calls. That's good enough to have real conversations.
I attended VAPI Con earlier this year, and a lot of the discussion centered on how interruptions and turn detection are the next frontier in making voice agents smoother conversationalists. Knowing when to speak is a hard problem even for humans, but when you listen to a lot of voice agent calls, the friction point right now tends to be either interrupting too often or waiting too long to respond.
The major players are clearly working on this. Deepgram announced a new SOTA (Flux) for turn detection at the conference. Feels like an area where we'll see even more progress in the next year.
I think interruptions had better be the top priority. I find text LLMs rage inducing with their BS verbiage that takes multiple prompts to reduce, and they still break promises like one sentence by dropping punctuation. I can't imagine a world where I have to listen to one of these things.
From my personal experience building a few AI IVR demos with Asterisk in early 2025, testing STT/TTS/inference products from a handful of different vendors, a reliable maximum latency of 2-3 seconds sounds like a definite improvement. Just a year ago I saw times from 3 to 8 seconds even on short inputs rendering short outputs. One half of this is of course over-committed resources. But clearly the executional performance of these models is improving.
Been experimenting with having a local Home Assistant agent include a qwen 0.5B model to provide a quick response to indicate that the agent is "thinking" about the request. It seems to work ok for the use case, but it feels like it'd get really repetitive for a 2 way conversation. Another way to handle this would be to have the small model provide the first 3-5 words of a (non-commital) response and feed that in as part of the prompt to the larger model.
How? Businesses will use this to justify removing what few actual human support staff they have left. Nobody, and I mean it, nobody calls customer support because they want to talk to a computer. It’s the last resort for problems that usually can’t be accomplished via the already existing technical flows available via computer.
The expectation of customer support lines is that customers want to speak to humans. It isn't just the fact that these are semantics that aren't written anywhere and are open to change, because by using a human-like voice agent on a customer support line, you are pretending that that is a human, which is a scam or fraud.
If you really believe that the support can be good, then use a robotic text to speech, don't pretend it's a human. And make it clear to users that they are talking to a human, phone is a protocol that has the semantic that you speak to a human. Use something else.
The bottom line is that you have clients that registered under the belief that they could call a phone number and speak to a human, businesses are performing a short-term switcheroo at the expense of their clients, it's a scam.
It is funny you mention Microsoft s customer support because it is a publicly known issue at this point that I'd you are a Microsoft employee or a v dash, the first level of support you talk to is basically something you have to overcome to get any help at all.
Please don't. I had a talk with a shitty AI bot on a Fedex line. It's absolute crap. Just give me a 'Type 1 for x, type 2 for y'. Then I don't need to guess what are the possibilities.
Voice-controlled phone systems are hugely rage-inducing for me. I am often in loud setting with background chatter. Muting my audio and using a touchtone keypad is so much more accurate and easy than having to find a quiet place and worrying that somebody is going to say something that the voice response system detects.
One problem is once you’re in deep building a phone IVR workflow beyond X or Y (yes, these are intentional), callers don’t care about some deep and featured input menu. They just mash 0 or pick a random option and demand a human finish the job and transfer them - understandably.
When you’re committed to phone intent complexity (hell), the AI assisted options are sort of less bad since you don’t have to explain the menu to callers, they just make demands.
The problem here is that if it's something a voice assistant can solve, I can solve it from my account. I'm calling because I need to speak to an actual human.
Im in this business, and used to think the same. It turns out this is a minority of callers. Some examples:
- a client were working does advertising in TV commercials, and a few percent of their calls is people trying to cancel their TV subscriptions, even though they are in healthcare
- in the troubleshooting flow for a client with a physical product, 40% of calls are resolved after the “did you try turning it off and on again” step.
- a health insurance client has 25% of call volume for something that is available self-service (and very visible as well), yet people still call.
- a client in the travel space gets a lot of calls about: “does my accommodation include X”, and employees just use their public website to answer those questions. (I.e., it’s clearly available for self-service)
One of the things we tend to prioritize in the initial conversation is to determine in which segment you fall and route accordingly.
(reposting because something ate your newlines, I've added comments in line)
Im in this business, and used to think the same. It turns out this is a minority of callers. Some examples:
- a client were working does advertising in TV commercials, and a few percent of their calls is people trying to cancel their TV subscriptions, even though they are in healthcare
I guess these are probably desperate people who are trying to get to someone, anyone. In my opinion, the best thing people can do is get a really good credit card and do a charge back for things like this.
- in the troubleshooting flow for a client with a physical product, 40% of calls are resolved after the “did you try turning it off and on again” step.
I bought a Chinese wifi mesh router and it literally finds a time between two am and five am and reboots itself every night, by default. You can turn this behavior off but it was interesting that it does this by default.
- a health insurance client has 25% of call volume for something that is available self-service (and very visible as well), yet people still call.
In my defense, I've been on the other side of this. I try to avoid calling but whenever I use self service, it feels like ny settings never stick and always switch back to what they want the next billing cycle. If I have to waste time each month, you have to waste time each month.
- a client in the travel space gets a lot of calls about: “does my accommodation include X”, and employees just use their public website to answer those questions. (I.e., it’s clearly available for self-service)
These public websites are regularly out of date. Someone who is actually on site confirm that yes, they have non smoking rooms or ice machines that aren't broken is valuable.
One of the things we tend to prioritize in the initial conversation is to determine in which segment you fall and route accordingly.
I’m honestly surprised it hasn’t been more prevalent yet. I still get call centre type spam calls where you can hear all the background noise of the rest of the call centre.
I assume it's to make it seem like an actual call center rather than a scam. I recently got two phone scam attempts (credit card related) that sounded exactly like this.
I built a voice AI stack and background noise can be really helpful to a restaurant AI for example. Italian background music or cafe background is part of the brand. It’s not meant to make the caller believe this is not a bot but only to make the AI call on brand.
You can call it what ever you like, but to me this is deceptive.
Where is the difference between this and Indian support staff pretending to be in your vicinity by telling you about the local weather? Your version is arguably even worse because it can plausibly fool people more competently.
Also, it only takes one legitimate collect call from a jail from a loved one and now I'm all in favor of reform in our jail system.
No, it does not cost over thirty dollars to allow someone accused to call their loved ones. We pay taxes. I want my government to use the taxes and provide these calls for free.
One easy way to build voice agents and connect them to Twilio is the Pipecat open source framework. Pipecat supports a wide variety of network transports, including the Twilio MediaStream WebSocket protocol so you don't have to bounce through a SIP server. Here's a getting started doc.[1]
(If you do need SIP, this Asterisk project looks really great.)
Pipecat has 90 or so integrations with all the models/services people use for voice AI these days. NVIDIA, AWS, all the foundation labs, all the voice AI labs, most of the video AI labs, and lots of other people use/contribute to Pipecat. And there's lots of interesting stuff in the ecosystem, like the open source, open data, open training code Smart Turn audio turn detection model [2], and the Pipecat Flows state machine library [3].
Disclaimer: I spend a lot of my time working on Pipecat. Also writing about both voice AI in general and Pipecat in particular. For example: https://voiceaiandvoiceagents.com/
I developed a stack on Cloudflare workers where latency is super low and it is cheap to run at scale thanks to Cloudflare pricing.
Runs at around 50 cents per hour using AssemblyAI or Deepgram as the STT, Gemini Flash as LLM and InWorld.ai as the TTS (for me it’s on par with ElevenLabs and super fast)
The baseline configurations all note <2s and <3s times. I haven't tried any voice AI stuff yet but a 3s latency waiting on a reply seems rage inducing if you're actually trying to accomplish something.
Is that really where SOTA is right now?
I've generally observed latency of 500ms to 1s with modern LLM-based voice agents making real calls. That's good enough to have real conversations.
I attended VAPI Con earlier this year, and a lot of the discussion centered on how interruptions and turn detection are the next frontier in making voice agents smoother conversationalists. Knowing when to speak is a hard problem even for humans, but when you listen to a lot of voice agent calls, the friction point right now tends to be either interrupting too often or waiting too long to respond.
The major players are clearly working on this. Deepgram announced a new SOTA (Flux) for turn detection at the conference. Feels like an area where we'll see even more progress in the next year.
I think interruptions had better be the top priority. I find text LLMs rage inducing with their BS verbiage that takes multiple prompts to reduce, and they still break promises like one sentence by dropping punctuation. I can't imagine a world where I have to listen to one of these things.
From my personal experience building a few AI IVR demos with Asterisk in early 2025, testing STT/TTS/inference products from a handful of different vendors, a reliable maximum latency of 2-3 seconds sounds like a definite improvement. Just a year ago I saw times from 3 to 8 seconds even on short inputs rendering short outputs. One half of this is of course over-committed resources. But clearly the executional performance of these models is improving.
Been experimenting with having a local Home Assistant agent include a qwen 0.5B model to provide a quick response to indicate that the agent is "thinking" about the request. It seems to work ok for the use case, but it feels like it'd get really repetitive for a 2 way conversation. Another way to handle this would be to have the small model provide the first 3-5 words of a (non-commital) response and feed that in as part of the prompt to the larger model.
Absolutely not.
500-1000ms is borderline acceptable.
Sub-300ms is closer to SOTA.
2000ms or more means people will hang up.
play "Just a second, one moment please <sounds of typing>".wave as soon as input goes quiet.
ChatGPT app has a audio version of the spinner icon when you ask it a question and it needs a second before answering.
Just try Gemini Live on your phone. That's state of the art
Microsoft Foundry's realtime voice API (which itself is wrapping AI models from the major players) has response times in the milliseconds.
Perhaps you didnt read that these are "production-ready golden baselines validated for enterprise deployment."
How does their golden nature not dissuade these concerns for you?
No, there are models with sub-second latency for sure
Sesame was the fastest model for a bit. Not sure what that team is doing anymore, they kind of went radio silent.
https://app.sesame.com/
What is the application of this that makes anything better for anyone? All I can think of is more spammers, scammers, horrible customer support lines.
Good customer support lines? Is there a reason why it can't provide good support. I often use chatgpt's voice function.
How? Businesses will use this to justify removing what few actual human support staff they have left. Nobody, and I mean it, nobody calls customer support because they want to talk to a computer. It’s the last resort for problems that usually can’t be accomplished via the already existing technical flows available via computer.
The expectation of customer support lines is that customers want to speak to humans. It isn't just the fact that these are semantics that aren't written anywhere and are open to change, because by using a human-like voice agent on a customer support line, you are pretending that that is a human, which is a scam or fraud.
If you really believe that the support can be good, then use a robotic text to speech, don't pretend it's a human. And make it clear to users that they are talking to a human, phone is a protocol that has the semantic that you speak to a human. Use something else.
The bottom line is that you have clients that registered under the belief that they could call a phone number and speak to a human, businesses are performing a short-term switcheroo at the expense of their clients, it's a scam.
Microsoft costumer support saar
It is funny you mention Microsoft s customer support because it is a publicly known issue at this point that I'd you are a Microsoft employee or a v dash, the first level of support you talk to is basically something you have to overcome to get any help at all.
> What is the application of this?
spammers, scammers and horrible customer support lines.
This opens up new possibilities for interactive phone services. Retro-futuristic for sure.
That seems like bad news for Allison. Though I know she already had some TTS voices available, so many not.
I've created Asterisk Codex Skill, but turns out there is ten seconds timeout for scripts
Please don't. I had a talk with a shitty AI bot on a Fedex line. It's absolute crap. Just give me a 'Type 1 for x, type 2 for y'. Then I don't need to guess what are the possibilities.
Voice-controlled phone systems are hugely rage-inducing for me. I am often in loud setting with background chatter. Muting my audio and using a touchtone keypad is so much more accurate and easy than having to find a quiet place and worrying that somebody is going to say something that the voice response system detects.
One problem is once you’re in deep building a phone IVR workflow beyond X or Y (yes, these are intentional), callers don’t care about some deep and featured input menu. They just mash 0 or pick a random option and demand a human finish the job and transfer them - understandably.
When you’re committed to phone intent complexity (hell), the AI assisted options are sort of less bad since you don’t have to explain the menu to callers, they just make demands.
What if the goal is to keep gaslighting you until you give up your demands?
Most voice agents for large companies are a calculated game to deter customers from expensive humans as we know, but not always.
Sort of like how Jira can be a streamlined tool or a prison of 50-step workflows, it's all up to the designer.
you bought something from the wrong company, and you arent gonna get helped by phone, bot, or person
The problem here is that if it's something a voice assistant can solve, I can solve it from my account. I'm calling because I need to speak to an actual human.
Im in this business, and used to think the same. It turns out this is a minority of callers. Some examples:
- a client were working does advertising in TV commercials, and a few percent of their calls is people trying to cancel their TV subscriptions, even though they are in healthcare - in the troubleshooting flow for a client with a physical product, 40% of calls are resolved after the “did you try turning it off and on again” step. - a health insurance client has 25% of call volume for something that is available self-service (and very visible as well), yet people still call. - a client in the travel space gets a lot of calls about: “does my accommodation include X”, and employees just use their public website to answer those questions. (I.e., it’s clearly available for self-service)
One of the things we tend to prioritize in the initial conversation is to determine in which segment you fall and route accordingly.
(reposting because something ate your newlines, I've added comments in line)
Im in this business, and used to think the same. It turns out this is a minority of callers. Some examples:
- a client were working does advertising in TV commercials, and a few percent of their calls is people trying to cancel their TV subscriptions, even though they are in healthcare
I guess these are probably desperate people who are trying to get to someone, anyone. In my opinion, the best thing people can do is get a really good credit card and do a charge back for things like this.
- in the troubleshooting flow for a client with a physical product, 40% of calls are resolved after the “did you try turning it off and on again” step.
I bought a Chinese wifi mesh router and it literally finds a time between two am and five am and reboots itself every night, by default. You can turn this behavior off but it was interesting that it does this by default.
- a health insurance client has 25% of call volume for something that is available self-service (and very visible as well), yet people still call.
In my defense, I've been on the other side of this. I try to avoid calling but whenever I use self service, it feels like ny settings never stick and always switch back to what they want the next billing cycle. If I have to waste time each month, you have to waste time each month.
- a client in the travel space gets a lot of calls about: “does my accommodation include X”, and employees just use their public website to answer those questions. (I.e., it’s clearly available for self-service)
These public websites are regularly out of date. Someone who is actually on site confirm that yes, they have non smoking rooms or ice machines that aren't broken is valuable.
One of the things we tend to prioritize in the initial conversation is to determine in which segment you fall and route accordingly.
Thx, forgot to double enter.
Well, the future is here: https://www.youtube.com/watch?v=HbDnxzrbxn4
I welcome the spam calls from our asterisk overlords.
I was more thinking I could add it to my Asterisk server to honey-pot the spam callers into an infinite time waster cycle.
"Hello, this is Lenny" - well known Asterisk configuration from 20 years ago.
I’m honestly surprised it hasn’t been more prevalent yet. I still get call centre type spam calls where you can hear all the background noise of the rest of the call centre.
Is the background noise real, or is it also AI-generated to make you think that it's a human?
The background noise is a recording for sure, no AI needed, just a background noise audiofile in a loop would do.
Why though? It adds nothing positive, it only makes me sure it is a scam call.
I assume it's to make it seem like an actual call center rather than a scam. I recently got two phone scam attempts (credit card related) that sounded exactly like this.
I built a voice AI stack and background noise can be really helpful to a restaurant AI for example. Italian background music or cafe background is part of the brand. It’s not meant to make the caller believe this is not a bot but only to make the AI call on brand.
You can call it what ever you like, but to me this is deceptive.
Where is the difference between this and Indian support staff pretending to be in your vicinity by telling you about the local weather? Your version is arguably even worse because it can plausibly fool people more competently.
you actually answer unknown callers?
Yes. I own a business.
Also, it only takes one legitimate collect call from a jail from a loved one and now I'm all in favor of reform in our jail system.
No, it does not cost over thirty dollars to allow someone accused to call their loved ones. We pay taxes. I want my government to use the taxes and provide these calls for free.
Yes. Sometimes it's a legit call. Not often, though.
Example of legit calls: the pizza delivery guy decided to call my phone instead of ringing the bell, for whatever reason.
I worked door dash for a couple of days and there were multiple people who wrote in all caps to not ring the door bell. Why? I have no idea.
Can I connect this to Twilio
One easy way to build voice agents and connect them to Twilio is the Pipecat open source framework. Pipecat supports a wide variety of network transports, including the Twilio MediaStream WebSocket protocol so you don't have to bounce through a SIP server. Here's a getting started doc.[1]
(If you do need SIP, this Asterisk project looks really great.)
Pipecat has 90 or so integrations with all the models/services people use for voice AI these days. NVIDIA, AWS, all the foundation labs, all the voice AI labs, most of the video AI labs, and lots of other people use/contribute to Pipecat. And there's lots of interesting stuff in the ecosystem, like the open source, open data, open training code Smart Turn audio turn detection model [2], and the Pipecat Flows state machine library [3].
[1] - https://docs.pipecat.ai/guides/telephony/twilio-websockets [2] - https://github.com/pipecat-ai/pipecat-flows/ [3] - https://github.com/pipecat-ai/smart-turn
Disclaimer: I spend a lot of my time working on Pipecat. Also writing about both voice AI in general and Pipecat in particular. For example: https://voiceaiandvoiceagents.com/
The problem with PipeCat and LiveKit (the 2 major stacks for building voice ai) is the deployment at scale.
That’s why I created a stack entirely in Cloudflare workers and durable objects in JavaScript.
Providers like AssemblyAI and Deepgram now integrate VAD in their realtime API so our voice AI only need networking (no CPU anymore).
let me get this straight, you are storing convo threads / context in DOs?
e.g. Deepgram (STT) via websocket -> DO -> LLM API -> TTS?
This is good stuff.
In your opinion, how close is Pipecat + OSS to replacing proprietary infra from Vapi, Retell, Sierra, etc?
I developed a stack on Cloudflare workers where latency is super low and it is cheap to run at scale thanks to Cloudflare pricing.
Runs at around 50 cents per hour using AssemblyAI or Deepgram as the STT, Gemini Flash as LLM and InWorld.ai as the TTS (for me it’s on par with ElevenLabs and super fast)
Do you have anything written up about how you're doing this? Curious to learn more...
Technically yes, twilio has sip trunks.