Realtime Voice AI Essentials

> Realtime voice AI lets a user have a natural, low-latency spoken conversation with an AI system — think voice assistants, AI phone agents, or in-car assistants. This notebook covers how these pipelines are built, the latency constraints that shape every design decision, and where they're actually used.

What is Realtime Voice AI?

Realtime voice AI is the class of systems that hold a live, spoken, back-and-forth conversation with a user — not a "record a message, get a text reply, have it read aloud" experience, but continuous audio in, continuous audio out, with the responsiveness of a phone call.

That responsiveness is the entire engineering challenge. Human conversation tolerates roughly 200–500ms of response latency before it starts feeling unnatural; a system built by chaining a batch speech-to-text step, an LLM call, and a batch text-to-speech step will typically blow past that, because each stage waits for the previous one to fully finish.

Two Architectures

1. Cascaded (pipeline) architecture Three separate models, streaming into each other:

Microphone → Speech-to-Text (streaming) → LLM (streaming) → Text-to-Speech (streaming) → Speaker

Each stage starts processing before the previous stage has fully finished — the LLM can start generating a response from partial transcript, and TTS can start speaking the first sentence while the LLM is still generating the rest. This is the more common and more controllable architecture: you can swap any component, add moderation between stages, and reuse your existing text-based LLM logic. 2. Speech-to-speech (native) architecture A single multimodal model takes audio in and produces audio out directly, without an intermediate text transcript as a hard boundary.

Microphone → Multimodal Model (audio-in, audio-out) → Speaker

This can achieve lower latency and preserve paralinguistic information (tone, emotion, pacing) that gets lost when everything is forced through a text transcript. It's less mature and harder to instrument (there's no clean text layer to log, moderate, or fine-tune against as easily).

Core Concepts

Streaming everything — audio in, transcript tokens, LLM tokens, and audio out are all processed as continuous streams, not discrete request/response calls

Voice Activity Detection (VAD) — detects when the user starts and stops speaking, so the system knows when to start listening and when to start responding

Interruption handling (barge-in) — the user needs to be able to interrupt the AI mid-sentence, which means the system must be able to stop generation and playback immediately, not just queue the interruption

Turn-taking — deciding when a pause means "the user is done talking" versus "the user is just thinking," which is genuinely hard to get right

Transport — realtime audio typically runs over WebRTC (low-latency, built for media) rather than plain HTTP, which isn't designed for continuous bidirectional streams

Common Use Cases

AI phone agents — customer support, appointment scheduling, outbound calling

Voice assistants — in-app or in-device assistants for hands-free interaction

Language practice / tutoring — conversational partners that need natural back-and-forth

In-car and accessibility assistants — contexts where hands and eyes aren't free for a screen

Meeting/call copilots — real-time transcription and assistance during live calls

Practical Example: A Minimal Cascaded Session (Sketch)

This illustrates the shape of a cascaded pipeline session, not a production implementation:

async def handle_voice_session(audio_stream):
    async for transcript_chunk in speech_to_text_stream(audio_stream):
        if is_end_of_turn(transcript_chunk):
            full_utterance = transcript_buffer.finalize()

            async for text_chunk in llm.stream(full_utterance):
                # start speaking before the LLM has finished generating
                async for audio_chunk in text_to_speech_stream(text_chunk):
                    await send_to_speaker(audio_chunk)

        elif user_started_speaking_again():
            # barge-in: stop current playback immediately
            await stop_playback()
            transcript_buffer.reset()

Several providers (e.g. OpenAI's Realtime API, ElevenLabs Conversational AI) now expose this entire loop — VAD, streaming STT/LLM/TTS or native speech-to-speech, and interruption handling — as a single managed API, which is usually the pragmatic starting point rather than assembling the pipeline from raw components.

Design Considerations

Latency budget — every added stage (moderation, RAG lookup, tool call) eats into the ~500ms budget; each one needs to either be fast enough to fit or be designed to run concurrently with speech generation

Cost — realtime audio models/APIs are typically priced well above equivalent text-only usage

Tool use mid-conversation — calling a tool (e.g. "look up my order") introduces latency the user will notice; systems often use a filler phrase ("let me check that...") to bridge the gap

Fallback behavior — network jitter or a slow tool call needs a graceful degradation path, not dead air

Testing — realtime voice systems are notoriously hard to test automatically; latency, interruption handling, and turn-taking often need real listening tests, not just unit tests

Key Takeaways

Latency, not raw quality, is the defining constraint of realtime voice AI — everything is designed around staying under ~500ms

Cascaded (streaming STT → LLM → TTS) is the more common, more controllable architecture; native speech-to-speech trades control for lower latency and richer paralinguistic signal

Interruption handling and turn-taking are as important to "feeling natural" as raw response quality

Managed realtime APIs (OpenAI Realtime, ElevenLabs Conversational AI, etc.) are usually the pragmatic starting point over building the pipeline from scratch

Resources

OpenAI Realtime API docs: https://platform.openai.com/docs/guides/realtime

WebRTC overview: https://webrtc.org