The Voice AI Stack

Building a voice assistant requires three components: Speech-to-Text (STT), an LLM for reasoning, and Text-to-Speech (TTS).

Speech-to-Text Options

  • Whisper (OpenAI): High accuracy, 99 languages, open-source.
  • Deepgram: Real-time streaming, low latency, conversation AI.
  • AssemblyAI: Good for meeting transcription with speaker diarization.
  • Google Speech-to-Text: Integrated with GCP, good for Android.

Text-to-Speech Options

  • ElevenLabs: Most natural voices, voice cloning, emotional range.
  • OpenAI TTS: Good quality, simple API, 6 voices.
  • Google WaveNet: Many languages, reasonable quality.
  • Coqui TTS: Open-source, self-hostable, voice cloning.

Architecture Patterns

### Simple Pipeline

1. Record audio → STT → text

2. Send text to LLM → response

3. Send response to TTS → play audio

### Streaming Pipeline (Lower Latency)

1. Stream audio chunks to STT in real-time.

2. As partial transcripts arrive, pre-warm the LLM.

3. Stream LLM response tokens to TTS incrementally.

4. Play audio chunks as they arrive.

Latency Optimization

  • Use streaming at every stage.
  • Pre-connect WebSocket connections.
  • Use edge functions for routing.
  • Target: < 1 second from user stop speaking to AI start responding.

Tools & Frameworks

  • Vocode: Open-source framework for voice agents.
  • LiveKit: Real-time audio/video infrastructure.
  • Pipecat: Daily's framework for voice AI pipelines.

Most production voice assistants follow a pipeline: Speech-to-Text (STT) → LLM processing → Text-to-Speech (TTS). The key design decision is whether to stream each stage or buffer the full response. Streaming reduces perceived latency dramatically—users hear the first word within 500ms instead of waiting 3–5 seconds for the full response.

Latency Budget

A good voice assistant should respond within 1–2 seconds end-to-end. Here’s a typical budget: STT adds 200–500ms, LLM first-token latency adds 300–800ms, and TTS adds 200–400ms. To stay under 2 seconds, use fast STT (Whisper large-v3-turbo or Deepgram Nova-2), a low-latency LLM (GPT-5-mini or Claude Haiku), and streaming TTS (ElevenLabs or PlayHT).

Common Pitfalls

  • Echo and barge-in: Without proper echo cancellation, the assistant hears itself and loops. Use a dedicated echo cancellation library or a platform that handles it.
  • Silence detection: Knowing when the user stopped talking is harder than it sounds. Voice Activity Detection (VAD) models like Silero VAD work well but need tuning for your environment.
  • Accent and dialect bias: Most STT models perform worse on non-standard accents. Test with diverse speakers and consider fine-tuning if your user base is diverse.