Every "voice AI" demo you see online follows the same script: someone types a prompt, a synthesized voice reads it back, and everyone pretends this is the future of customer service. Then you try to deploy it on a real phone line and discover that the demo skipped every hard problem.

Real phone calls have background noise, cross-talk, impatient callers who interrupt, cellular packet loss, and a 200ms latency ceiling before the caller assumes nobody is there and hangs up. The "hello world" of voice AI is trivially easy. The production version — the one that answers your business phone 24/7 and doesn't embarrass you — is a completely different engineering problem.

I built a voice AI receptionist that handles inbound calls for a small business in Marion, Ohio. It runs on a self-hosted Asterisk PBX on a $5/month LXC container, uses SIP trunking from Callcentric for the phone number, and routes to a Pipecat-based voice pipeline that does real-time speech-to-text, LLM reasoning, and text-to-speech — all with sub-second response latency. Here is how it works and what I learned building it.

Why Most Voice AI Products Fail in Production

Before getting into the architecture, it is worth understanding why most off-the-shelf voice AI solutions fall apart when you point a real phone number at them.

Latency kills conversations. Human conversational turn-taking has a natural gap of about 200-400ms. If your AI takes longer than ~800ms to start responding, callers perceive dead air. They say "hello?" again, which creates a feedback loop where the STT picks up the second "hello" and starts processing that instead of the original question. The caller gets frustrated and hangs up. Most cloud-based voice AI pipelines have 1.5-3 seconds of latency end-to-end, which is dead on arrival for real calls.

Interruption handling is non-negotiable. Real callers interrupt. They correct themselves mid-sentence. They say "actually, wait" and change topics. A voice AI that cannot handle barge-in — where the caller speaks over the AI's response — sounds robotic and frustrating. This requires full-duplex audio processing, not the half-duplex "wait for silence then respond" that most demos use.

Phone audio is terrible. SIP calls over PSTN use narrow-band codecs (G.711 at 8kHz, sometimes G.729 compressed). The audio quality is dramatically worse than the studio-quality microphone recordings that speech-to-text models are trained on. Background noise, speakerphone echo, Bluetooth compression, and cellular packet loss compound the problem. Your STT accuracy in production will be 10-20% worse than your benchmark numbers.

The IVR trap. Most businesses that deploy voice AI end up building an interactive voice response tree with extra steps. "Press 1 for sales, press 2 for support" becomes "say sales or support." This is not AI — it is a voice-controlled menu. A real voice AI agent should handle freeform conversation, extract intent from messy human speech, and take appropriate action without forcing the caller into a decision tree.

The Architecture: Asterisk + Pipecat + LLM

The system has three layers: the phone system (Asterisk PBX), the voice pipeline (Pipecat), and the brain (LLM with business context). Here is how they connect.

Layer 1: Asterisk PBX

Asterisk is the open-source telephone switch that has been running phone systems since 2001. It handles SIP registration, codec negotiation, call routing, voicemail, and all the PSTN plumbing. I run it in an LXC container on Proxmox — CT110 at 192.168.1.210 — with 512MB RAM and 1 CPU core. The entire phone system uses about 30MB of memory at idle.

; extensions.conf — inbound call routing
[from-external]
exten => s,1,Answer()
 same => n,Wait(0.5)
 same => n,Set(CHANNEL(language)=en)
 same => n,AGI(agi://127.0.0.1:4573)
 same => n,Hangup()

; Missed call handler — fires if AGI fails or caller hangs up
exten => h,1,System(/usr/local/bin/missed_call.sh ${CALLERID(num)} ${CDR(duration)})

The key decision here is using AGI (Asterisk Gateway Interface) instead of ARI or a SIP REFER. AGI gives the voice pipeline direct access to the audio channel — it can read and write audio frames in real time, which is critical for barge-in handling. ARI would work too, but adds a WebSocket layer that increases latency.

SIP trunking comes from Callcentric. The economics matter: a DID (phone number) costs about $1.50/month. Inbound calls are free. Outbound is $0.01-0.02/minute. For a small business that receives 10-30 calls per day, the total phone cost is under $5/month — less than a single month of any cloud voice AI platform.

Layer 2: Pipecat Voice Pipeline

Pipecat is the real-time voice framework that handles the streaming pipeline: audio in → speech-to-text → LLM → text-to-speech → audio out. The critical design constraint is that all of this must happen in a streaming fashion — you cannot wait for the caller to finish speaking, then process the whole utterance, then generate the whole response, then speak it. Each step must stream into the next.

# Voice pipeline configuration
pipeline = Pipeline([
    transport.input(),           # SIP audio frames (8kHz PCM)
    stt.DeepgramSTTService(      # Real-time transcription
        model="nova-2",
        encoding="linear16",
        sample_rate=8000,
        language="en",
        endpointing=300,         # ms of silence before finalizing
        interim_results=True,    # Stream partial transcripts
    ),
    llm_processor,               # Context-aware LLM (see Layer 3)
    tts.ElevenLabsTTSService(    # Neural voice synthesis
        voice_id="jennifer",
        model_id="eleven_turbo_v2_5",
        output_format="pcm_8000", # Match SIP codec
        optimize_streaming_latency=4,
    ),
    transport.output(),          # Back to SIP channel
])

The pipeline runs as a systemd service on the same container. Key tuning parameters:

Layer 3: LLM with Business Context

The LLM is not a generic assistant. It is a purpose-built receptionist with specific knowledge about the business, its services, pricing, hours, and staff. The system prompt is dense with operational detail — not "be helpful and polite" but "here are the 6 services we offer, their prices, the owner's name is Aaron, we service Marion and 7 surrounding towns, here is how to handle pricing questions."

This is the same approach I use for the AI sales agent on LuxuriousComputers — the LLM's personality and knowledge come from a carefully crafted system prompt, not from fine-tuning or RAG retrieval. For a receptionist handling 10-30 calls per day, the system prompt approach is simpler, more reliable, and easier to update than a vector database.

# System prompt structure for voice AI receptionist
SYSTEM_PROMPT = """You are the AI receptionist for {business_name}.
Your voice is warm, professional, and efficient. You speak in
SHORT SENTENCES — never more than 2 sentences before pausing
for the caller's response. Phone conversations are different
from text: be concise.

SERVICES AND PRICING:
{service_list_with_prices}

HOURS: {business_hours}

OWNER: {owner_name}, available for callbacks {callback_hours}

CALL HANDLING RULES:
1. Greet by name if caller ID matches a known customer
2. For service inquiries: confirm the service, give the price
   range, and offer to schedule
3. For emergencies: get their address and phone, tell them
   someone will call back within 15 minutes
4. For existing appointments: check the calendar and confirm
5. NEVER quote exact prices for custom work — say "typically
   $X to $Y depending on the situation" and offer a callback
6. If you cannot help: take their name and number, promise
   a callback within 2 hours

CRITICAL: You are on a PHONE CALL. Keep responses under 30 words.
Long responses cause callers to interrupt or hang up."""

The 30-word response limit is the single most important tuning parameter. In text chat, a 200-word response feels thorough. On a phone call, a 200-word response takes 45 seconds to speak, during which the caller has checked out, started talking to someone else, or hung up. Voice AI responses must be radically shorter than text AI responses.

Barge-In: The Hardest Problem in Voice AI

Barge-in is when the caller starts speaking while the AI is still talking. In a human conversation, this happens constantly — the listener says "yeah" or "right" to signal they are following, or they interrupt to correct a misunderstanding. A voice AI that cannot handle this sounds like a recording.

The implementation requires full-duplex audio processing. The pipeline must simultaneously:

  1. Stream TTS audio to the caller
  2. Listen for incoming audio from the caller
  3. Detect whether the incoming audio is speech (not background noise or echo)
  4. If speech detected: immediately stop TTS playback, cancel any queued audio, and start processing the new utterance
# Barge-in handler
class BargeInProcessor(FrameProcessor):
    def __init__(self):
        self._is_speaking = False
        self._energy_threshold = 0.02  # Calibrated for G.711

    async def process_frame(self, frame):
        if isinstance(frame, TTSStartedFrame):
            self._is_speaking = True

        elif isinstance(frame, TTSStoppedFrame):
            self._is_speaking = False

        elif isinstance(frame, AudioRawFrame) and self._is_speaking:
            energy = self._rms_energy(frame.audio)
            if energy > self._energy_threshold:
                # Caller is speaking over TTS — interrupt
                await self.push_frame(TTSStopFrame())
                await self.push_frame(LLMResetFrame())
                self._is_speaking = False

    def _rms_energy(self, audio_bytes):
        samples = np.frombuffer(audio_bytes, dtype=np.int16)
        return np.sqrt(np.mean(samples.astype(float) ** 2)) / 32768

The energy threshold (0.02) needs calibration per environment. Too low and background noise triggers constant false interrupts. Too high and the system ignores soft-spoken callers. In practice, I use a dynamic threshold that adapts to the noise floor of each call during the first 2 seconds of silence after answer.

Echo cancellation is the other critical piece. When the AI speaks, its audio feeds through the phone and comes back through the caller's microphone (especially on speakerphone). Without echo cancellation, the system detects its own voice as a barge-in and interrupts itself in an infinite loop. Asterisk provides hardware echo cancellation via DAHDI, but for SIP-only setups, you need software echo cancellation — typically WebRTC's AEC (acoustic echo cancellation) module or a custom implementation using the Speex library.

Missed Call Recovery: No Call Goes Unanswered

The voice AI answers instantly — no rings, no hold music, no "your call is important to us." But sometimes calls fail: the AGI process crashes, the LLM times out, or the caller hangs up before the AI finishes its greeting. Every one of these is a lost lead.

The missed call handler fires on the Asterisk h extension (hangup). It captures the caller's number and call duration, then triggers a notification pipeline:

#!/bin/bash
# missed_call.sh — fires on every hangup
CALLER=$1
DURATION=$2

# Skip if call lasted > 30 seconds (was answered and handled)
[ "$DURATION" -gt 30 ] && exit 0

# Push notification via ntfy (self-hosted)
curl -s -d "{\"topic\":\"business-missed-calls\",\"title\":\"Missed Call\",\"message\":\"From: $CALLER | Duration: ${DURATION}s\",\"priority\":4}" \
  https://ntfy.beamvideos.com/publish

# SMS notification via email-to-SMS gateway
for carrier_gw in txt.att.net tmomail.net vtext.com messaging.sprintpcs.com; do
  echo "Missed call from $CALLER (${DURATION}s)" | \
    mail -s "Missed Call" "OWNER_PHONE@$carrier_gw"
done

The notification pipeline uses the same pattern I deploy across all my Cloudflare Workers backends — self-hosted ntfy for push notifications plus email-to-SMS gateway broadcast for immediate owner alerting. No Twilio subscription, no monthly SMS fees. The gateway broadcast sends to all major carriers simultaneously; only the owner's actual carrier delivers.

The Latency Budget: Where Every Millisecond Goes

End-to-end latency — from when the caller finishes speaking to when they hear the first syllable of the AI's response — must be under 800ms. Here is how the budget breaks down in production:

Total: 630-830ms. Tight but viable. The key insight is that every component must stream. If any single step waits for the previous step to fully complete before starting, the budget explodes.

For comparison, here is what happens with a naive (non-streaming) implementation:

Model Selection: Speed vs. Intelligence

For the LLM step, model choice is constrained by the 250ms first-token budget. In practice, this limits you to:

I use Claude Haiku for the receptionist because it handles nuanced intent classification better than smaller models. When a caller says "I think my furnace might be making a weird noise, it started yesterday, should I be worried?" — Haiku correctly classifies this as a potential emergency, while an 8B model might treat it as a general inquiry. For a business receiving real calls, classification accuracy matters more than saving $0.001 per call.

Caller ID Integration: Personalized Greetings

When the system recognizes a returning caller, the greeting changes from "Thank you for calling [business]. How can I help you?" to "Hi [name], good to hear from you again. What can I do for you today?" This one change dramatically shifts the caller's perception from "I'm talking to a robot" to "they remember me."

The implementation uses a simple KV lookup keyed on the caller ID number. Every call logs the caller's name (extracted from the conversation) and their last interaction topic. On subsequent calls, this context is injected into the LLM's system prompt.

# Caller context injection
caller_id = event.headers.get("Caller-ID-Number", "unknown")
caller_ctx = await kv.get(f"caller:{caller_id}")

if caller_ctx:
    ctx = json.loads(caller_ctx)
    system_prompt += f"""

RETURNING CALLER: {ctx['name']}
Last called: {ctx['last_call']}
Previous topic: {ctx['last_topic']}
Notes: {ctx.get('notes', 'None')}

Greet them by name. Reference their previous interaction
naturally if relevant — e.g., "Did that furnace issue get
resolved?" Do NOT be creepy about it."""

IVR Fallback: When AI is Not the Answer

Not every call should go to the AI. Some callers want to speak to a human immediately. Some callers are spam. Some situations require human judgment (angry customer, legal issue, emergency that needs immediate dispatch).

The system detects these cases and routes appropriately:

Cost Breakdown: Self-Hosted vs. Cloud Voice AI

The total monthly cost for this system handling ~20 calls per day:

Total: ~$11.50/month.

For comparison, cloud voice AI platforms charge $0.10-0.50 per minute of conversation. At 20 calls/day averaging 2 minutes, that is $120-600/month — 10-50x the self-hosted cost. The cloud platforms include a nice dashboard and phone number provisioning, but the underlying technology is the same STT → LLM → TTS pipeline.

The self-hosted approach also means you own the caller data. No third party has recordings of your customer conversations. For businesses in regulated industries (healthcare, legal, financial), this is not optional — it is a compliance requirement.

What I Would Do Differently

Three things I learned building this that I would change on the next deployment:

1. Start with Deepgram's on-prem STT. The 50-80ms cloud latency is acceptable but not ideal. Deepgram offers a self-hosted container that runs on CPU and brings STT latency down to 10-20ms. For a system where latency is the primary constraint, trimming 50ms from STT buys you headroom for a larger LLM or slower TTS.

2. Use voice cloning from day one. I started with a stock ElevenLabs voice. Three weeks in, the business owner wanted the AI to sound more like their actual receptionist. Switching voices mid-deployment means every returning caller notices the change. If I had cloned the receptionist's voice from the start (a 30-second recording through F5-TTS or ElevenLabs Instant Voice Cloning), the transition would have been seamless.

3. Record and review the first 50 calls. Asterisk can record every call to WAV files with a single dialplan line: MixMonitor(${UNIQUEID}.wav). I did this retroactively after discovering that the AI was giving incorrect pricing for one service. If I had been reviewing calls from day one, I would have caught the system prompt error on call #3 instead of call #40.

The Result

The system has been running for several months. It answers every call within one ring, handles routine inquiries without human intervention, and escalates correctly when it should. The business owner gets push notifications for every call with a summary, and missed calls trigger SMS alerts within seconds.

The callers mostly do not realize they are talking to an AI — not because the voice is perfectly human (it is not), but because the AI responds quickly, answers their actual question, and does not waste their time. That is the bar for production voice AI: not passing a Turing test, but being more useful than a voicemail box.

Related Articles

AI SalesClaude

Why Your AI Chatbot Sucks (And How to Build One That Closes Sales)

The text-based counterpart: building an AI sales agent that handles objections, tracks cart state, and actually converts.
Self-HostedGPU

How I Run a 27B LLM on Consumer GPUs for $0/Month

The inference infrastructure that powers this voice pipeline — running Qwen 27B on commodity hardware.
CloudflareEdge

Cloudflare Workers as Your AI Backend

The notification and lead capture layer that voice calls route into — ntfy push, SMS broadcast, KV storage.