The Voice AI Stack

Building a voice assistant requires three components: Speech-to-Text (STT), an LLM for reasoning, and Text-to-Speech (TTS).

Speech-to-Text Options

  • Whisper (OpenAI): High accuracy, 99 languages, open-source.
  • Deepgram: Real-time streaming, low latency, conversation AI.
  • AssemblyAI: Good for meeting transcription with speaker diarization.
  • Google Speech-to-Text: Integrated with GCP, good for Android.

Text-to-Speech Options

  • ElevenLabs: Most natural voices, voice cloning, emotional range.
  • OpenAI TTS: Good quality, simple API, 6 voices.
  • Google WaveNet: Many languages, reasonable quality.
  • Coqui TTS: Open-source, self-hostable, voice cloning.

Architecture Patterns

Simple Pipeline

1. Record audio → STT → text

2. Send text to LLM → response

3. Send response to TTS → play audio

Streaming Pipeline (Lower Latency)

1. Stream audio chunks to STT in real-time.

2. As partial transcripts arrive, pre-warm the LLM.

3. Stream LLM response tokens to TTS incrementally.

4. Play audio chunks as they arrive.

Latency Optimization

  • Use streaming at every stage.
  • Pre-connect WebSocket connections.
  • Use edge functions for routing.
  • Target: < 1 second from user stop speaking to AI start responding.

Tools & Frameworks

  • Vocode: Open-source framework for voice agents.
  • LiveKit: Real-time audio/video infrastructure.
  • Pipecat: Daily's framework for voice AI pipelines.

Most production voice assistants follow a pipeline: Speech-to-Text (STT) → LLM processing → Text-to-Speech (TTS). The key design decision is whether to stream each stage or buffer the full response. Streaming reduces perceived latency dramatically—users hear the first word within 500ms instead of waiting 3–5 seconds for the full response.

Latency Budget

A good voice assistant should respond within 1–2 seconds end-to-end. Here’s a typical budget: STT adds 200–500ms, LLM first-token latency adds 300–800ms, and TTS adds 200–400ms. To stay under 2 seconds, use fast STT (Whisper large-v3-turbo or Deepgram Nova-2), a low-latency model tier (for example Claude Haiku 5.5 or GPT-6 Luna), and streaming TTS (ElevenLabs or PlayHT).

Common Pitfalls

  • Echo and barge-in: Without proper echo cancellation, the assistant hears itself and loops. Use a dedicated echo cancellation library or a platform that handles it.
  • Silence detection: Knowing when the user stopped talking is harder than it sounds. Voice Activity Detection (VAD) models like Silero VAD work well but need tuning for your environment.
  • Accent and dialect bias: Most STT models perform worse on non-standard accents. Test with diverse speakers and consider fine-tuning if your user base is diverse.

Latency Is the Whole Product

A voice assistant is judged on the pause between the end of your sentence and the start of its reply. Anything above roughly a second of silence reads as broken, and that budget must cover speech recognition, model inference, and the first chunk of speech synthesis. Optimising model quality while ignoring this end-to-end number is the most common way voice projects fail late and expensively.

The practical technique is to overlap the stages. Start synthesising as soon as the first sentence of the response exists, stream the recognition result rather than waiting for the final transcript, and let the user interrupt. Barge-in is not a nicety; without it, a misheard question traps the user in a monologue they cannot stop.

Everything Is Worse in Real Audio

Text benchmarks use clean input. Real input has background noise, overlapping speakers, regional accents, and the acoustics of a kitchen. Test with recordings from your actual users, in the environments they actually use, and measure word error rate per environment rather than as a single average. A system that is excellent in a quiet room and unusable in a car is unusable, and a single averaged number hides that completely.

Designing for Being Wrong

Speech recognition will mishear names, numbers, and domain terms, and the assistant must recover gracefully. Confirm before acting on anything irreversible, read back identifiers digit by digit, and always provide a way to correct without starting over. Silence after a misrecognition is worse than a clarifying question.

Model names were checked against the official documentation of OpenAI and Anthropic on 2026-11-10.