KI-Sprachassistenten entwickeln: Von Sprache zu Text und zurück

The Voice AI Stack

Building a voice assistant requires three components: Speech-to-Text (STT), an LLM for reasoning, and Text-to-Speech (TTS).

Speech-to-Text Options

• Whisper (OpenAI): High accuracy, 99 languages, open-source.
• Deepgram: Real-time streaming, low latency, conversation AI.
• AssemblyAI: Good for meeting transcription with speaker diarization.
• Google Speech-to-Text: Integrated with GCP, good for Android.

Text-to-Speech Options

• ElevenLabs: Most natural voices, voice cloning, emotional range.
• OpenAI TTS: Good quality, simple API, 6 voices.
• Google WaveNet: Many languages, reasonable quality.
• Coqui TTS: Open-source, self-hostable, voice cloning.

Architecture Patterns

### Simple Pipeline

1. Record audio → STT → text

2. Send text to LLM → response

3. Send response to TTS → play audio

### Streaming Pipeline (Lower Latency)

1. Stream audio chunks to STT in real-time.

2. As partial transcripts arrive, pre-warm the LLM.

3. Stream LLM response tokens to TTS incrementally.

4. Play audio chunks as they arrive.

Latency Optimization

• Use streaming at every stage.
• Pre-connect WebSocket connections.
• Use edge functions for routing.
• Target: < 1 second from user stop speaking to AI start responding.

Tools & Frameworks

• Vocode: Open-source framework for voice agents.
• LiveKit: Real-time audio/video infrastructure.
• Pipecat: Daily's framework for voice AI pipelines.