AI phone assistant technology: latency, speech recognition and interruptions

On the phone, silence is a signal. Whoever asks a question and still hears nothing after a moment says “Hello?” or repeats the sentence. That is exactly where the typical failure pattern of voice assistants begins: the caller starts talking just as the system's answer is starting up, both fall silent, both start again. After two or three such moments the conversation has lost its rhythm.

Suppose a business first tests an assistant in the office, with a clear voice, over Wi-Fi and in a quiet room. The demo is convincing because none of it resembles a real call. Out there, people phone from the building site, from the car, with headphones, with a child in the background. They speak quickly, break off sentences and give names that no dictionary knows.

This study puts the technology behind a voice assistant into context so that you can read offers and demos critically. It is a technical analysis, not project documentation, and therefore deliberately gives no measured values from deployments. It is about mechanisms: where time is lost, where understanding fails and which countermeasures have proven effective.

In the classic design, a voice assistant consists of three stages. Speech recognition converts audio to text, a language model decides what to answer, and speech synthesis turns that back into sound. Each stage needs time, and the times add up to the pause the caller hears. Added to this are transmission paths between the phone network, provider and servers.

It is important that the total pause does not consist of computing time alone. The system first has to recognise that the caller has finished. This endpoint detection decides whether an answer is given after a short breath or only after a longer silence. If it is too sensitive, the system interrupts the speaker. If it is too sluggish, a pause arises that nobody attributes to the model but to the business.

A second challenge is understanding under poor conditions. Phone calls are transmitted with limited bandwidth, mobile networks compress further, and background noise such as wind or machinery masks consonants. It is precisely consonants that carry the information in names and numbers. “Fünf” and “elf”, or “Meier” and “Maier”, then become a guess.

The third challenge lies in the dialogue itself. People interrupt, acknowledge with “mhm” and correct themselves mid-sentence. An assistant that can only speak and listen in turn feels like an answering machine with a read-aloud function. Good technology must be able to listen and speak at the same time and decide what is interference and what is intent.

Besides the cascade there are models that take in audio directly and output audio, without the intermediate step via text. They promise more natural intonation and shorter pauses, because no handover between stages is needed. Filler sounds and interruptions can also be handled more fluidly. The price is less control: what exactly is said is harder to check in advance.

The cascade is technically less elegant, but easier to steer. The text between the stages can be logged, filtered and checked against rules before it is spoken. That matters where an assistant must not invent prices or make commitments. You can also swap individual building blocks, for example change the voice without touching the model.

Which design fits depends on the use. For conversations with a clear flow, such as taking an appointment or a callback request, the cascade with good streaming is usually manageable and traceable. For free, longer conversations, a speech-to-speech model can seem noticeably more natural. The decision should rest on the conversation pattern and not on the marketing claim about the architecture.

Hybrid forms are also possible. Some setups use a fast recognition model on the listening side and let a larger model re-check only the difficult passages. Others place pre-generated recordings in front of fixed announcements such as the greeting and the privacy notice, which start without delay and free up computing time for the first real answer.

A sober caveat: the model landscape changes quickly. Statements about which provider currently delivers the lowest latency go out of date within months. What lasts is the method: run the same recorded test conversations through both designs and compare pauses, comprehension errors and controllability.

The most effective lever against pauses is streaming. Instead of waiting for each stage to finish, every stage passes its interim result along immediately. Speech recognition delivers partial text continuously, the language model starts answering as soon as it knows enough, and speech synthesis speaks the first sentence while the second is still being written. The times overlap instead of adding up.

Another trick helps, one that people use themselves: a short acknowledgment or a transitional phrase while the system is working. “One moment, let me check” bridges an appointment lookup in which a calendar has to be queried. What matters is to use such fillers sparingly and honestly. If you say “Just a moment” after every question, you sound like a hold queue, and the trick wears out.

A further measure is to keep processing paths short. The closer the telephony connection, recognition and model run to one another, the less network distance is involved. The length of the system prompt and the number of tool calls per answer also affect the pause. An assistant that queries three systems for every piece of information will be slower than one that keeps the few necessary details on hand.

The goal is not a particular number but a coherent rhythm. Whether a pause is perceived as natural or disruptive also depends on the type of question: after a simple yes-no question, people expect an immediate answer; after a question that requires calculating, they tolerate some thinking time.

The technical term for interruptions is barge-in. While the assistant is speaking, recognition stays active. When the caller starts talking, the output stops and the system processes what was said. It sounds simple, but it is tricky, because not every noise is an interruption. A cough, a passing truck or a “mm-hm” should not cut the announcement short.

In practice, systems solve this with thresholds. Only when recognizable speech is present for a short duration does it count as an interruption. A pure acknowledgment such as “yes” or “right” can be deliberately ignored or treated as agreement. This fine-tuning differs for every use case: a construction site needs more robust filters than a quiet office.

Echo is a stumbling block of its own. When the caller is on speakerphone, the system hears its own voice on the return channel and can interrupt itself. Echo cancellation is therefore part of the basic equipment of the telephony connection. You should deliberately provoke it in tests, for example with a hands-free kit in a car.

Then there is the question of how long the system waits after the last word. Someone dictating a phone number pauses between groups of digits for longer than in a normal sentence. An assistant with a fixed waiting time cuts them off in the middle of the number. Better is endpoint detection that takes the state of the conversation into account: if the system has just asked for a number, it waits longer; for a yes-no question, it answers promptly.

After an interruption, the system must know what the caller still heard. If the sentence with the price was cut off halfway, the model must not assume the information got through. Good implementations remember up to which word the assistant spoke and pick up from there.

Technically, an assistant can be thought of in layers that can each be tested individually. At the bottom is the telephony connection, which accepts calls and hands over audio. Above it sits speech processing, with recognition and synthesis. Next comes dialogue control, with the language model, rules and conversation state. At the top are the connections to the calendar, phone system or CRM, which supply information and enter appointments.

The advantage of this view: errors can be traced to a layer. If the assistant hears “Müller” instead of “Möller”, the cause is recognition or the audio, not the model. If the answer is right but stressed wrongly, the cause is synthesis. And if the system invents an appointment, the cause is the control layer, which did not make the calendar data binding.

Quality also depends on what the system must not do on its own. Dialogue control should know firm limits: no binding prices, no legal or medical statements, and handover to a person when in doubt. These limits belong in rules and checks, not just in a prompt, because a prompt is a request, not a barrier.

Finally, logging and measurement points belong in every layer. Timestamps at the transitions show where pauses arise. Stored recognition texts make it possible to review misunderstandings afterwards. Call recordings touch on data protection and consent. You should clarify the rules for this with your data protection officer or adviser before launch; this is orientation, not legal advice.

Dialects and accents are not a niche topic but everyday life on the phone. Modern speech recognition understands most regional accents surprisingly well, but errors rise with strong dialect, fast speech or mixed languages. A blanket statement about recognition rates would not be credible, because they depend on the provider, the language variant and the audio quality. The only thing that counts is a test with the voices of your own callers.

Three things help with noise. Noise suppression cleans the audio before recognition, which works well for constant noise such as engines, but less well for changing voices in the background. Second, the system can honestly ask for clarification: “I’m having trouble hearing you, could you say that again?” Third, with a persistently poor connection, it should offer a callback early rather than drag the conversation out.

Proper names, streets, phone numbers and house numbers are especially error-prone. A fixed confirmation loop works well here: the system repeats the detail in a different form and has it confirmed. For numbers, it reads them digit by digit; for names, it offers spelling, including with the help of the German spelling alphabet if the caller is willing.

A comparison with data the system already has is also valuable. If someone gives a street, it can be checked against a list of places. For existing customers, the phone number can suggest the name. This saves follow-up questions but carries a trap: treating a guess as fact confirms errors. The comparison should therefore always be phrased as a question, not as a statement.

Realistically, a well-built assistant can be expected to handle calls with a clear goal reliably: requesting an appointment, arranging a callback, giving opening hours, taking down a request. The narrower the conversation goal, the better the results. Where open-ended advice is needed, the error rate rises, and the path to a person must be short.

The benefit can be measured with figures you collect yourself. These include the rate of conversations that reach their goal without a handover, the number of drop-offs after the first answer, how often callers have to repeat themselves because they were not understood, and the quality of the details captured, such as names and phone numbers. Add to that the pause between the end of the caller’s utterance and the start of the answer, measured in the logs.

There is no credible promise about recognition rates, pause lengths or satisfaction before testing. Too many factors lie outside the system: network quality, device, environment, way of speaking. Speech technology does keep improving, but improvements arrive in the building blocks, not automatically in your business. Your results only change when you re-measure and readjust.

Voice choice belongs in this consideration as well. A voice that speaks too fast or too ceremoniously causes more repeat questions, even when recognition works flawlessly. So choose the voice not by its sound in the studio but by how intelligible it is over a mobile network. Listen to candidates through a basic phone speaker profile and with slight noise, and have people from your circle write down numbers and street names that the voice reads out.

One honest point more: an assistant does not replace a relationship. Callers who are upset or have a sensitive matter want a person. Success also shows in how well the handover works, not just in how many calls the assistant completes on its own.

Demo sentences prove little because they are spoken the way the system likes to hear them. Tests with conversation patterns from your own business are meaningful. Collect typical requests from everyday work, ideally from notes, callback slips and your staff’s recollections. For data protection reasons, real recordings should only be used with consent and for a defined purpose; otherwise, staff re-enact the calls.

A good test set deliberately includes difficult cases: callers with a dialect, with background noise, on speakerphone, someone who changes the subject mid-sentence, someone who gives a long double-barreled name, someone who says “Hold on, I need to check something.” Conversations the assistant is not supposed to handle at all are just as important, such as complaints or questions outside its remit.

The evaluation looks not only at whether a correct result ends up at the end, but also at the path to it. How often did it have to ask again, where did a conspicuous pause occur, at what point was there a false interruption? A conversation that succeeds but gets out of rhythm twice is a finding, not a success.

One practical check concerns repeatability. After a test, change only one thing, such as the waiting time of the endpoint detection, and replay the same conversations. Only then can you see whether a change helps. If you turn several dials at once, you will not know afterwards what worked, and you risk improvement and deterioration masking each other.

Testing continues after launch. Listening to a sample of conversations every week is the most effective quality measure there is. This is where the best improvements come from: an added technical term, an adjusted confirmation, an earlier callback offer.

A voice assistant is good when it behaves unobtrusively in conversation: no long pauses, no talking over people, no misheard names that only come to light at the end. Behind this is less magic than careful engineering and a lot of testing. Anyone who understands this can compare offers better and probe promises for substance.

The voice is part of it, but it is not the main thing. A calm, clearly understandable voice with subtle emphasis works better on the phone than an imitation that feigns humanity. Tell callers that they are speaking with a digital assistant. This is honest and, in our assessment, widely accepted when the conversation then works. You should clarify the legal disclosure obligations in advance.

Anyone who buys or commissions this technology should ask three questions. How is the pause measured, and where in the system? How does the assistant behave with interruptions, echo and noise? And what happens when it does not understand something, after the second attempt? Answers that are concrete and verifiable point to a mature system. Evasive answers that only stress naturalness are a warning sign.

The next step is small: write down ten typical calls to your business, with the most difficult details, and have them replayed. In an hour, you will see which questions the technology has to solve. We are happy to help turn this into a test plan and to assess together whether a voice assistant is a good idea for your business, and where it is not.