How Voice AI Works
A non-technical explanation of the technology behind your AI voice agents.
The Technology Behind Your AI Receptionist
You don't need to understand how an engine works to drive a car — and you don't need to understand AI internals to run a successful agency. But having a basic grasp of how the technology works helps you set expectations with clients, troubleshoot issues, and sell with confidence.
Note: Think of your AI voice agent like a very well-trained receptionist. It listens carefully, understands what the caller needs, responds naturally, and follows the instructions you've given it. The difference? It never takes a lunch break, never calls in sick, and handles every call with the same consistent quality at 2 AM as it does at 2 PM.
The Call Flow — What Happens in Real Time
When someone calls your client's AI-powered phone number, here's exactly what happens — in about the time it takes to blink:
Incoming Call Arrives
The caller dials the phone number assigned to your client. Twilio (our telephony provider) receives the call and routes it to the AI Agency Unlocked platform instantly.
Agent Identity Loaded
The platform identifies which client and agent this number belongs to. It loads the business profile, custom greeting, FAQs, knowledge base documents, and all configuration — in milliseconds.
Greeting Delivered
The AI agent speaks the custom greeting using ElevenLabs voice synthesis. The voice is natural, warm, and human-sounding — not robotic. Example: "Thanks for calling Sunrise Dental, this is Sarah. How can I help you today?"
Caller Speaks — Speech Recognized
As the caller talks, their speech is converted to text in real time using advanced speech recognition. This handles accents, background noise, and natural speech patterns with high accuracy.
AI Understands and Responds
The transcribed text is sent to Claude Haiku 4.5 by Anthropic, which understands the caller's intent and generates an appropriate response based on the business profile, FAQs, and knowledge base. This is where the "intelligence" lives — the AI reasons about what the caller needs and crafts a helpful, natural reply.
Response Spoken Aloud
The AI's text response is converted back to speech using ElevenLabs, matching the voice and tone configured for this agent. The caller hears a natural, conversational response — as if they're talking to a real person.
Post-Call Processing
After the call ends, the platform processes the conversation: generating a summary, identifying the caller's intent, scoring quality and sentiment, extracting any data (names, emails, phone numbers), recording actions taken (bookings, messages), and creating or updating the contact record. All of this appears in the portal within seconds.
Why ElevenLabs + Claude Haiku 4.5?
We chose the best-in-class technology for each piece of the puzzle:
| Component | Technology | Why We Chose It |
|---|---|---|
| Voice synthesis | ElevenLabs | Most natural-sounding AI voices available. Callers frequently can't tell it's AI. |
| Intelligence | Claude Haiku 4.5 (Anthropic) | Best at following instructions, staying on-topic, and handling nuanced conversations. |
| Telephony | Twilio | Industry standard for phone infrastructure. Reliable, global, carrier-grade. |
What Your AI Agent CAN Do
- Answer questions about the business using the knowledge base and FAQs
- Book appointments (when Cal.com is connected)
- Collect caller information — name, phone, email, reason for calling
- Handle multiple calls simultaneously (never a busy signal)
- Work 24 hours a day, 7 days a week, 365 days a year
- Speak naturally with appropriate pauses, tone, and conversational flow
- Recognize returning callers and reference their history
- Follow business hours — different behavior during open vs. closed hours
- Take messages when the business is closed or can't handle a request
- Transfer calls to a human when needed
What Your AI Agent CANNOT Do
- Make outbound calls (it only answers incoming calls)
- Process payments or take credit card numbers
- Access external systems unless a connector is configured
- Guarantee 100% accuracy — it can occasionally misunderstand or give imperfect answers
- Handle highly emotional or crisis situations (e.g., medical emergencies) the way a trained human can
- Understand every accent or heavy background noise perfectly
- Perform physical tasks (obviously!) — it can schedule a plumber visit, but it can't fix the pipe
Warning: Always be transparent with your clients about what the AI can and cannot do. Setting realistic expectations upfront leads to happier clients and fewer support issues. The AI is an exceptional receptionist — not a replacement for every human interaction.
How It Gets Smarter Over Time
Your AI agent doesn't "learn" on its own from calls — but it gets better as you refine its configuration. When you notice the agent struggling with a particular question, add it to the FAQs. When a client's services change, update the knowledge base. Think of it like training a new employee: the better your documentation, the better the performance.
Was this page helpful?