How to Build a Custom AI Phone Agent for Your Business in Under an Hour

Most businesses lose calls in two places: after-hours, and during the day when the owner is on a job and the line is busy. Call centers are expensive, inflexible, and still miss nuance. What actually works is a custom AI phone agent — one that knows your services, speaks in your tone, and passes qualified leads straight to your calendar.

The good news: you can build a working version in under an hour. Here is the stack and the exact order to set it up.

The stack

You need three things:

Twilio — for the phone number, call routing, and webhook infrastructure.
ElevenLabs — for the AI voice. This is what makes the agent sound human instead of robotic.
A small bridge server — to connect Twilio's voice stream to the AI, handle your custom prompt, and manage tool calls (like sending a summary or creating a calendar event).

Total running cost for a small business: roughly $30–70/month, depending on call volume. Compare that to $500–1,500/month for a traditional answering service.

Step 1: Buy a Twilio number

Sign up for Twilio, buy a local phone number in your area code, and enable voice on it. If you want to keep your existing business number, you can forward calls to the Twilio number when you are busy or after-hours.

Tip

Test with a new number first. Once the agent works, port or forward your main line. This avoids breaking your existing flow during setup.

Step 2: Create an ElevenLabs agent

In ElevenLabs, create a conversational agent. Choose a voice that matches your brand — calm and professional for HVAC, warm and reassuring for pest control. The voice matters more than most people think.

Write a system prompt that tells the agent exactly how to behave. Include:

Your business name and services
How to greet the caller
What questions to ask to qualify the lead
Operating hours and location
When to offer a callback vs. booking directly

Step 3: Build the bridge server

The bridge is a lightweight server that sits between Twilio and ElevenLabs. It streams audio from the caller, sends it to ElevenLabs for processing, and streams the AI voice back in real time.

It also handles the "brain" — custom logic beyond the prompt. For example:

Sending a call summary to your email or Slack
Creating a calendar hold after booking
Writing the lead details to a simple database or spreadsheet

This does not need to be complex. A Node.js or Python server with WebSocket support is enough.

Step 4: Connect Twilio to your bridge

In Twilio, configure the incoming webhook for your number to point to your bridge server's URL. When a call comes in, Twilio forwards the audio stream to your server. Your server forwards it to ElevenLabs, gets the AI response, and sends audio back.

Step 5: Add tool calls

This is where the agent becomes useful instead of just conversational. Set up "tools" the agent can invoke during a call:

Send summary — after the call ends, email or text you the caller's name, issue, and availability.
Calendar check — if you use Google Calendar or Cal.com, the agent can ask what time works and create a hold.
CRM write — add the lead to a simple Airtable or Notion database.

Step 6: Test, iterate, deploy

Call your own number. See how the agent sounds. Check if it asks the right questions. Fix the prompt where it gets confused. Most improvements come from prompt edits, not engineering changes.

When you are happy with it, flip your business line to forward to the AI number during off-hours or when you are busy.

What this replaces

Voicemail that never gets checked
Expensive answering services that still miss context
Lost leads because nobody picked up at 6:17 PM
Scattered notes from callback requests

Common mistakes to avoid

Over-scripting the prompt. Give the agent boundaries, not a word-for-word script. A rigid script breaks the moment the caller asks something unexpected.

Skipping the follow-up automation. The call is only half the job. If the summary does not reach you in a format you actually check, the agent is just a fancy voicemail.

Ignoring voice quality. A slightly unnatural voice ruins trust. Test multiple ElevenLabs voices and pick the one that fits your business.

Want the complete kit?

The AI Voice Agent Builder's Kit includes the bridge server code, setup steps, tool-call examples, cost breakdown, and troubleshooting reference — everything you need to go from zero to live AI answering in one afternoon.

If you would rather we build and manage it for you, ShipClean handles the entire stack — number, voice, prompt tuning, and follow-up — for home-service businesses. No code required.

Browse the kit →

Building your own agent is simpler than it sounds. The hard part is not the technology — it is knowing the right order to set it up, and what to say in the prompt. Once you get those two things right, the rest is wiring.

For more on how we use AI to keep home-service businesses from losing leads, see 7 AI workflows for home services and the exact AI lead response system we ship with ShipClean.