What changes on a phone line
A browser demo has clean, wideband audio, a stable connection and a user who expects to talk to a bot. A phone call has narrowband audio, background noise, speakerphones, people who talk over the agent and callers who want a human. The agent's model may be the same; the plumbing around it is not.
Narrowband audio affects recognition. Names, email addresses and numbers are harder to catch at phone quality, so build confirmation into the conversation: read back the phone number, spell the name, or text a link for anything long. Test with real calls from mobile phones in noisy rooms, not just a headset at your desk.
Whatever LLM and voice stack you use, you need a way to get call audio to it and its audio back, a way to transfer and hang up, and a number that carriers trust.
Option one: media-stream WebSockets
Most programmable voice providers can fork a call's audio to a WebSocket you host. The provider handles the phone network; you receive audio frames as messages and send audio back the same way.
Formats vary by provider. Twilio's Media Streams, for example, sends and expects 8 kHz mu-law audio, base64-encoded inside JSON messages, which is the same narrowband quality as an ordinary phone call. Telnyx offers bidirectional streaming over RTP with a choice of codecs, including wideband options. Resampling to and from what your speech models expect is your job, and it adds a small amount of processing time.
WebSockets are the faster path to a working agent. You keep your existing code shape, and call control (transfer, hang up, play audio) is done through the provider's API alongside the stream.
Option two: SIP
SIP is the signaling protocol the phone network and business phone systems speak. Connecting your voice stack over SIP means it appears as a phone endpoint: the call is delivered to your SIP server, and audio flows as RTP media.
SIP gives you more control over codecs, standard transfer methods and integration with existing phone systems. It also brings more to operate: SIP servers, media handling, firewall and NAT traversal, and the many ways SIP implementations disagree with each other. Teams with telephony experience or an existing voice platform often prefer it. Teams coming from the AI side usually start with WebSockets.
Latency, turn-taking and barge-in
People notice delay on the phone quickly. Research comparing conversations across ten languages found that speakers everywhere tend to avoid long silences and overlapping speech between turns, so a pause of a second or more before the agent responds feels wrong.
Latency in a voice agent is the sum of several stages: carrier and network transit, end-of-speech detection, speech recognition, the language model's first token, voice generation's first audio, and the trip back. Streaming at every stage, starting voice generation before the full reply is written, and hosting close to the media servers all help.
End-of-speech detection is a quiet source of delay. Waiting for a fixed silence before responding is simple but slow; cutting it too short makes the agent interrupt people mid-sentence. Measure the full round trip on real calls, not in the browser.
Callers interrupt. When they do, the agent has to stop talking at once and listen. That means detecting caller speech while your own audio is playing, then discarding whatever audio you've already sent but hasn't played yet.
Providers support this in different ways. In Twilio's bidirectional Media Streams, for example, a clear message empties buffered audio, and mark messages tell you which audio actually played. Track what the caller heard, so the agent's context doesn't include sentences that were cut off.
Transfers to people
Every production agent needs to hand calls to a person. A blind transfer simply connects the caller elsewhere. A warm transfer lets the agent or a summary brief the staff member first, which is much better for the caller.
Over SIP, transfers are commonly done with the REFER method, defined in RFC 3515. Over WebSockets, you ask the provider's call-control API to dial the staff member and bridge the call. Either way, decide what happens when nobody answers: return to the agent, take a message, or go to voicemail.
Plan for failure on both sides. If your agent's WebSocket drops mid-call, the caller should land somewhere useful, such as a fallback message or a staff line, rather than dead air.
Also decide what the staff member sees. A transfer that arrives as an unknown number with no context is barely better than a cold call. Passing the caller's name, the reason for the call and a short summary, either whispered before the bridge or shown on screen, is what makes a warm transfer worth the effort.
Recording consent and AI disclosure
If you record or transcribe calls, consent rules apply. Some US states require all parties to consent to a recording, and when a call crosses state lines the stricter rule is the safe one to follow. A short announcement at the start of the call is the usual approach.
AI rules also apply. The FCC ruled in 2024 that AI-generated voices are artificial voices under the TCPA, which brings consent requirements for outbound calls, and it has proposed further disclosure rules. Telling callers they're speaking with an AI assistant is a sensible default regardless.
Numbers and caller ID trust
If the agent makes outbound calls, such as reminders or callbacks, carriers judge those calls by the number and its signing. Under the FCC's STIR/SHAKEN rules, the originating provider signs each call with an attestation level. Calls from numbers the provider can't tie to a known business get lower attestation and are more likely to be labeled as spam.
For texts sent alongside calls, such as a booking link mid-call, the number also needs carrier registration for business texting. Plan for both before the first business goes live.
Running it for many businesses
A single-business agent can hard-code its number, prompt and transfer target. A platform can't. Configuration has to be looked up at the moment the call arrives, from the dialed number, and that lookup sits on the latency path, so cache it. You'll need:
- A mapping from each inbound number to its business, configuration and knowledge.
- Per-business registration, call signing and caller name, attached number by number.
- Isolation, so one business's data never reaches another's conversations.
- Concurrency limits and capacity planning for busy hours across all businesses at once.
- Per-business usage metering for billing.
- Per-business recording and retention settings.
How Wirebird handles this
Wirebird's bring-your-own-agent option points a tenant's AI handling at your WebSocket or SIP endpoint. You receive call audio in both directions, or live transcripts as events if you'd rather work with text, and you can ask Wirebird to transfer, hold, hang up or send a text mid-call.
Wirebird keeps the parts that aren't AI: numbers, per-business registration, call signing, consent announcements, recording and transfers to staff. You can run your agent for some tenants and Wirebird's AI receptionist for others.
Sources
- Twilio: Media Streams WebSocket messages
- Telnyx: Media streaming
- RFC 3515: The Session Initiation Protocol (SIP) Refer Method
- Stivers et al. (2009), Universals and cultural variation in turn-taking in conversation, PNAS
- Reporters Committee for Freedom of the Press: Reporter's Recording Guide
- FCC Declaratory Ruling FCC 24-17 on AI-generated voices under the TCPA (summary, Mayer Brown)
- Federal Register: Implications of AI Technologies on Protecting Consumers From Unwanted Robocalls (FCC 24-84)
- FCC: Caller ID authentication (STIR/SHAKEN) order and implementation, FCC 21-62