Self-Hosted Real-Time Voice AI Agents

Problem

The operator wanted AI voice agents for customer service and IVR, but recordings, transcripts, and prompts had to stay on private infrastructure. Callers also expect a natural conversation: fast replies, the ability to interrupt, and switching language mid-call.

Approach

I am building a streaming Pipecat pipeline where each model runs as its own GPU service behind an OpenAI-compatible API, so providers can be swapped per agent. Operators configure an agent once (persona, cloned voice, knowledge documents, guardrails, languages) and every call builds a fresh pipeline from that configuration. In parallel I am generalizing it into a reusable multi-tenant accelerator with a management UI, and evaluating voice cloning and lip-synced avatars for reusable Accenture AI assets.

How It Works

1. 1
   ConnectWeb and mobile app over WebRTC or WebSocket, plus telephony proofs of concept: phone calls bridged through Asterisk SIP with a custom audio serializer (G.711, Opus, AMR-WB), and WhatsApp calling.
2. 2
   ListenNoise reduction and Silero VAD detect speech; a turn-detection model decides when the caller has finished speaking.
3. 3
   TranscribeWhisper compiled with TensorRT-LLM, constrained to the call languages, with a cleanup step for hallucinated output.
4. 4
   ThinkQwen3-32B (AWQ) on vLLM streams the reply. Tools can end the call or switch language, and a second pass enforces guardrails and strips formatting that should not be spoken.
5. 5
   SpeakA GPT-SoVITS voice fine-tuned for Indonesian, with a custom phoneme (G2P) frontend, streams audio back; voices can be cloned from reference audio, and each language has its own TTS branch.
6. 6
   Record & reviewThe caller can interrupt at any moment (barge-in). Transcripts and recordings are stored for the operator dashboard.

What I Built

• The orchestrator service and per-call Pipecat pipeline, including tool calls and mid-call language switching.
• Model services: Whisper STT on TensorRT-LLM, Qwen3-32B (AWQ) on vLLM, and GPT-SoVITS TTS fine-tuned for Indonesian with a custom G2P phoneme frontend.
• Telephony proofs of concept: an Asterisk SIP/RTP bridge with codec handling for phone audio, and WhatsApp calling.
• GPU services on AWS ECS with Terraform: GPU-metric autoscaling, CloudWatch monitoring, and shared model storage.
• The reusable accelerator: a React management UI (agents, voices, knowledge, guardrails, conversations, model playgrounds) with swappable cloud or self-hosted providers.
• Locust load tests measuring STT, LLM, and TTS latency and throughput.

Impact

Still in progress. The voice agent already runs end to end on private GPUs over web and app channels, with phone and WhatsApp calling proven in proofs of concept, and its building blocks feed reusable Accenture AI assets. I wrote nearly all of the client deployment’s code.