Voice Agents in Production: Latency, Turn-Taking, Interruptions, and Recovery

A text agent can pause for six seconds and still look thoughtful.

A voice agent pauses for six seconds and sounds dead.

Then the user says, "Hello?" just as the agent starts speaking. The agent hears the interruption, talks over it, submits the old appointment time anyway, and cheerfully confirms the wrong result.

The model may have reasoned correctly. The conversation failed as a real-time system.

Voice changes the engineering problem. You are no longer moving discrete messages through a request-response loop. You are coordinating continuous audio, uncertain turn boundaries, overlapping speakers, streaming generation, tool calls, user corrections, playback buffers, and durable business state.

A production voice agent must be correct in two dimensions at once:

1. Task correctness: Did it perform the right action?
2. Interaction correctness: Did the conversation remain understandable, responsive, and recoverable while it did so?

Optimizing one without the other produces a demo, not a service.

---

TL;DR

• Choose the audio architecture before tuning prompts.
• Measure time to useful audio, not merely model response time.
• Treat listening, reasoning, tool work, generation, and playback as separate states.
• An interruption must stop playback, update conversational history, and decide what happens to in-flight work.
• Keep consequential tools and credentials on a trusted server boundary.
• Confirm exact names, dates, numbers, and irreversible actions before committing.
• Evaluate recordings, event streams, tool results, and final application state together.
• Design explicit paths for silence, disconnects, noisy audio, corrections, and human handoff.

---

Choose the architecture first

There are three common voice-agent architectures. Each makes a different tradeoff.

Chained speech-to-text -> text agent -> text-to-speech

This is the easiest architecture to inspect. You get explicit transcripts, can reuse an existing text agent, and can independently swap recognition, reasoning, and speech components.

The cost is latency and a less fluid conversation. Each stage waits on the previous one unless you carefully stream and speculate.

Use it when:

• exact transcripts matter
• the task agent already exists
• compliance requires inspecting text before speech
• conversational naturalness is less important than control

Native speech-to-speech

A realtime model consumes and produces audio directly. It can preserve tone and timing information that a text bridge loses, and it usually feels more natural.

The tradeoff is a more complex event model. You still need transcripts for audit, tool calls for work, and application state for truth.

Use it when:

• low latency is central to the product
• interruptions and emotional nuance matter
• the conversation itself is the primary interface

Live conversation plus a delegated backend agent

One component manages listening, speaking, and short conversational turns while a backend agent handles long reasoning and tools. The live layer can acknowledge the user, ask clarifying questions, and stay responsive while work continues.

This split is increasingly useful in 2026. OpenAI's current voice guidance describes a similar division: a live conversation layer can keep talking while backend work proceeds, while the application still owns permissions, tools, and saved progress. (OpenAI voice agents guide)

Use it when:

• tool calls can take seconds
• the backend workflow already exists
• you need natural timing without giving the conversational layer unrestricted authority

The best architecture is not the one with the fewest boxes. It is the one whose failure modes your team can observe and control.

Build a latency budget, not a latency wish

"The model is fast" does not mean the caller hears a fast answer.

Perceived latency includes:

speech detection
+ end-of-turn decision
+ network transit
+ reasoning or delegation
+ tool execution
+ first audio generation
+ playback buffering
= time to useful audio

Instrument each stage separately. A two-second delay may come from conservative voice-activity detection rather than model reasoning. A fast acknowledgment may hide a ten-second wait for the answer that matters.

Track at least:

• connection establishment
• speech start and stop detection
• end-of-turn commit
• first transcript token
• backend request start
• tool start and completion
• first useful result
• first output audio received
• first output audio played
• final audio played

Measure median and tail latency. One smooth demo says little about a poor 95th percentile on mobile networks.

OpenAI's voice evaluation guidance similarly recommends measuring task outcome, audible response timing, unwanted silence, overlap, interruptions, and session reliability independently. (Evaluating voice agents)

"Fast" and "responsive" are not the same

If a tool needs four seconds, you have three choices.

Say nothing

Efficient, but the user may think the call dropped.

Fill the gap

"Great question. Let me take a look at that for you now."

This can feel responsive once. Repeated filler quickly becomes irritating.

Give a meaningful progress cue

"I'm checking the two open orders now. This may take a few seconds."

This is usually better because it sets an expectation and names the work. Keep it short and truthful. Never claim an action has started until the backend accepted it.

Preambles should be driven by expected operation duration and user need, not generated reflexively before every tool call.

Turn detection is a product decision

The agent needs to decide when the user has finished speaking. There is no universally correct timeout.

Simple voice-activity detection

Audio energy falls below a threshold for long enough, so the turn ends. It is fast and predictable but cuts off users who pause to think.

Semantic turn detection

A model estimates whether the utterance sounds complete. This can handle "I need a flight to..." differently from "I need a flight to Denver," but may wait longer and introduces another model decision.

Push to talk

The user explicitly marks boundaries. It sacrifices conversational flow for control and works well in noisy, high-consequence environments.

Hybrid controls

Use automatic detection with a visible hold-to-talk or "done" control as a recovery path.

Tune by context. A restaurant booking agent can use a brisker turn boundary than a health-intake assistant where users pause, search for dates, or become emotional.

Current realtime APIs expose both server and semantic voice-activity modes, along with controls for automatic responses and interruption behavior. (Realtime conversations) The API setting is the easy part. Choosing the experience is the design work.

Interruption is a distributed state transition

When a user speaks over the agent, "stop talking" is only the visible requirement.

The system must coordinate:

1. detect new user speech
2. stop generation if appropriate
3. clear or stop queued playback
4. truncate the assistant message to what the user actually heard
5. preserve the new input
6. decide whether in-flight tools should continue, pause, or cancel
7. produce a response based on the corrected state

If the transcript retains audio that was generated but never played, the model believes it said something the user did not hear. The next turn becomes subtly incoherent.

WebRTC and SIP systems may know how much audio has played and can handle truncation at the server. With a WebSocket audio player, the client often owns that bookkeeping. OpenAI's realtime documentation explicitly separates interruption from truncation for this reason. (Realtime interruption and truncation)

Treat playback position as state.

Decide what an interruption means for tools

Suppose the user says:

Move my reservation to seven.

The agent starts an availability check. The user interrupts:

Actually, make that seven-thirty tomorrow.

Should the old check continue?

It depends on the operation.

In-flight work	Interruption policy
Read-only lookup	Continue if result may still help; mark inputs stale
Expensive search	Cancel when supported
Reversible reservation hold	Cancel or replace using the same workflow ID
Payment or irreversible write	Never infer cancellation; reconcile exact status

Attach every delegated job to a revision of user intent:

{
  "conversationId": "call_91",
  "intentRevision": 4,
  "task": "check_availability",
  "arguments": {
    "date": "2026-10-24",
    "time": "19:30"
  }
}

When a result returns, compare its revision with the current intent. A stale result can inform the conversation but must not silently commit the old request.

Separate conversational state from business state

The transcript is not the database.

Spoken conversation contains corrections, hypotheticals, partial identifiers, and acknowledgments. Durable business state should contain only validated, committed facts.

Maintain separate records for:

• raw and normalized transcript
• current interpreted intent
• unresolved slots or questions
• pending tool operations
• committed application changes
• confirmations the user actually heard

This prevents "I might want Tuesday" from becoming a Tuesday booking and prevents an assistant's unplayed confirmation from being treated as informed consent.

Confirm what is hard to hear

Voice is weak at exact strings. Names, email addresses, confirmation codes, dates, amounts, and similar-sounding numbers deserve special treatment.

Use selective confirmation:

• read back consequential values
• group long identifiers into chunks
• ask for spelling only when confidence is low
• show a visual companion when available
• distinguish speech-recognition confidence from business validation
• require explicit confirmation immediately before irreversible actions

Do not read an entire form back after every turn. Confirm the minimum information whose error would matter.

For financial or legal actions, the confirmation should bind to a structured pending action:

Agent: I have a $125 refund to card ending 2048, reason "service failure." Should I submit it?
User: Yes.

If the amount or destination changes, the approval is no longer valid.

Keep tools and credentials behind the server

A browser or phone client should not receive durable tool credentials. Use a trusted server connection for private tool execution, authorization, policy checks, and audit logging.

Realtime platforms may support a sideband control connection so media flows directly while the application server observes events and executes sensitive work. (OpenAI server-side controls)

The security rules remain the same as for text agents:

• least-privilege credentials
• user-bound authorization
• server-side argument validation
• approval gates for consequential actions
• auditable tool receipts
• isolation between tenants and sessions

Voice does not lower the permission standard because the interface feels casual.

Design for silence and disconnects

Silence can mean:

• the user is thinking
• the microphone is muted
• the network stalled
• the agent is waiting on a tool
• audio generation failed
• playback is blocked
• the call disconnected

Do not use one timeout for all of them.

Track session liveness, media liveness, backend progress, and user-idle state separately. Then respond appropriately:

• user idle: gentle prompt, then wait
• tool delay: meaningful progress cue
• media failure: attempt reconnection or switch channel
• backend failure: explain and offer retry or handoff
• disconnect after a write: persist the result and provide another confirmation channel

If a call drops after booking succeeds but before the spoken confirmation, the workflow is not "failed." It is complete with an undelivered notification.

Human handoff must carry the conversation, not just the call

Transferring audio without context forces the user to repeat everything.

Give the human:

• concise issue summary
• verified identity state
• structured fields and confidence
• actions already taken
• pending or ambiguous operations
• transcript excerpt around the problem
• reason for escalation
• user sentiment only when responsibly inferred and labeled

Tell the caller what will happen before the transfer. If the handoff queue is long, offer a callback or asynchronous continuation instead of trapping the user in silence.

The same principles in the Human Handoff Playbook apply, but voice adds timing and emotional continuity.

Evaluate what the user actually experienced

A transcript-only test misses half the system.

For each evaluation run, retain the privacy-appropriate combination of:

• input audio or a protected reference
• output audio
• transcripts with timestamps
• speech start and stop events
• generation and playback boundaries
• interruptions and truncations
• tool requests, results, and receipts
• final application state
• connection and error events

Score four layers.

Task outcome

Was the correct real-world action completed once?

Conversational timing

Was the first useful response timely? Were there awkward silences, overlaps, or ignored interruptions?

Speech quality

Were names and numbers understandable? Did recognition hold across accents, code switching, background noise, and poor microphones?

Recovery

Could the agent handle corrections, tool errors, disconnects, and handoff without losing state?

Use synthetic speech for repeatable baseline tests, recorded human speech for acoustic diversity, and simulated multi-turn callers for interruptions and changing requirements. Then include human listening. A waveform metric cannot tell you that the agent sounds impatient during a bereavement call.

Production metrics

Track:

• task completion and verified side-effect accuracy
• time to first useful audio, p50 and p95
• unwanted-silence duration
• overlap and successful barge-in rate
• false interruption rate
• correction rate for names, dates, and numbers
• stale tool-result discard rate
• dropped and incomplete sessions
• human handoff rate and context completeness
• confirmation mismatch rate
• cost per completed conversation

Segment by transport, network class, language, device, and use case. Aggregate voice metrics hide the users having the worst experience.

Implementation checklist

• Select chained, native realtime, or delegated architecture intentionally.
• Define stage-level latency budgets.
• Instrument generation and playback separately.
• Choose and test a turn-detection policy.
• Implement interruption plus transcript truncation.
• Version user intent for asynchronous tools.
• Separate conversation from committed business state.
• Confirm consequential exact values.
• Keep private tools behind a trusted server.
• Design distinct silence and disconnect paths.
• Transfer structured context during human handoff.
• Evaluate audio, events, tools, and final state together.

The bottom line

Voice agents do not fail only by saying the wrong thing. They fail by speaking too late, talking over a correction, confirming an action that never committed, or losing the thread when a tool takes longer than a human pause.

The best voice systems are not merely fast models attached to microphones. They are carefully coordinated real-time applications with durable work behind them.

Build the conversation and the transaction as two connected systems. Then make sure an interruption cannot tear them apart.

---

Further reading

• OpenAI: Voice Agents
• OpenAI: Realtime Conversations
• OpenAI: WebRTC
• OpenAI: Server-Side Voice Controls