Commstate Docs

Calls — Real-time voice plan

What is shipped today, what is missing, and the order to build it in.

Today a call reaches a human by ringing their mobile, and the AI cannot answer at all because there is no media worker. This plan replaces the first and supplies the second, without changing the routing, state, billing or provider work already in place.

What already exists

Worth being precise, because it determines how much is actually left.

piece state
Call model, state machine, routing, queues, agents shipped
Plivo adapter: XML, signatures, numbers, compliance shipped
<Dial><User>sip:…</User></Dial> for non-PSTN agents already rendered — see PlivoXmlRenderer::dialAgent
call_agents.endpoint_type of webrtc / sip / pstn already in the schema
Media session handshake: token, Redis context, /calls/media/* shipped, nothing connects to it
Transfer with context (transcript carried to the agent) shipped
Turn-based voice assistant (record → transcribe → chat → TTS) shipped in the mobile app; push-to-talk, not a conversation
STT/TTS providers (Whisper, OpenAI, Google, Voxtral) shipped, file-based — not streaming

Two gaps: nobody registers a browser or app endpoint, and nothing holds the audio socket.

The decision: one media domain, not two

Plivo terminates the PSTN leg and can also terminate the agent leg through its Browser and Android SDKs. An agent registered as a Plivo endpoint is dialled with the same <Dial><User> the platform already emits.

That keeps every participant in one media domain. The alternative — a WebRTC system such as LiveKit for the agent side — means bridging PSTN and WebRTC for every transferred call: two failure domains, and either a SIP trunk or a relay process holding both ends for the call's duration.

LiveKit is not ruled out, it is deferred. It earns its place when a call needs more than two participants, video, screen share, or supervisor listen-in — none of which the current product asks for. Because the media plane sits behind CALLS_MEDIA_WS_URL and the /calls/media/* contract, the transport can change later without touching the control plane. That was the point of the separation.

Architecture

   PSTN caller ──── Plivo ─────┐
                               │
   Browser agent ─ Plivo       ├──  <Dial><User>  bridges them natively
     (Browser SDK, WebRTC)     │    no relay, no SIP trunk, no second system
                               │
   Android agent ─ Plivo ──────┘
     (Android SDK)

   AI ──── Plivo <Stream> ──── Pipecat worker
                                 │
                                 ├─ realtime model (OpenAI Realtime / Gemini Live)
                                 │  or VAD → STT → LLM → TTS
                                 │
                                 └─ /calls/media/{session}/attach|transcript|
                                    heartbeat|transfer|close   ← already built

The AI and the human are reached the same way — the AI through a <Stream>, the human through a <Dial> — so a transfer between them is the existing redirect, not a new mechanism.

Phases

Ordered so each one is independently useful and independently revertible.

Phase 1 — Agents answer in the browser

The change you asked for, and the cheapest: no new service, no new infrastructure.

  • Provision a Plivo endpoint per agent (username, password, alias) through the existing NumberProviderContract account.
  • Store it on call_agents: endpoint_type: 'webrtc', endpoint_address: sip:<username>@phone.plivo.com. The column already exists.
  • A <CallBar> component in the web app using the Plivo Browser SDK: register on login, show incoming calls, answer / hang up / mute / hold, and DTMF.
  • Presence already exists — going online registers the endpoint, and an agent without one is already skipped by the router.

Nothing in routing or transfer changes. bridgeToAgent already renders <User>sip:…</User> for a webrtc endpoint.

Risk: browsers need microphone permission and a live WebSocket; an agent whose tab is closed is unreachable. Presence handles this — the endpoint deregisters and isAvailable() already requires an endpoint address.

Phase 2 — Agents answer in the Android app

Same shape, Plivo's Android SDK instead of the Browser SDK.

The one real constraint: the app is Expo, and the Plivo Android SDK is native. That needs a development build with a config plugin, not Expo Go. Worth knowing before it is scheduled — it changes how the app is built and distributed, which is a bigger change than the calling code itself.

Phase 3 — The AI media plane

A Pipecat worker, deployed as its own service.

  • Plivo WebSocket transport — Pipecat handles the 20 ms framing, audio conversion, interruption and DTMF, so none of that is written here.
  • Two pipeline modes behind one interface:
    • realtime model — OpenAI Realtime or Gemini Live, speech to speech, which removes the STT→LLM→TTS hops entirely and is the difference between a bot that feels laggy and one that does not;
    • composed — VAD → STT → LLM → TTS, for providers without a realtime model, and cheaper per minute.
  • Reaches the platform through the endpoints that already exist: attach for context, transcript per utterance, heartbeat so the reaper can tell a dead worker from a quiet one, transfer when the model asks for a human, close at the end.
  • The agent, its knowledge base and its tools are the ones already in services/ai-agent. Voice is a transport for the same agent, not a second one.

The worker names no vendor

The worker does not know what Mistral is. attach hands it a provider profile — brain, ears and mouth, each with a type, base URL, key, model and (for speech) a voice — resolved from the tenant's own AI module registry by RealtimeProfileService. The worker builds its pipeline from that.

attach ──► { llm: {type, base_url, api_key, model},
             stt: {type, base_url, api_key, realtime_model},
             tts: {type, base_url, api_key, model, voice_id} }

Three things follow, and they are why it is shaped this way:

  • swapping Voxtral for Deepgram, or Mistral for a self-hosted model, is a settings change rather than a redeploy;
  • the same registry already serves chat and the turn-based phone path, so a tenant configures a provider once and all three use it;
  • a tenant who has configured nothing gets a worker that refuses the session, rather than one quietly running on another tenant's account.

Only the vendor adapters inside the worker know a vendor, exactly as CallsPlivo is the only thing in the call path that knows Plivo.

Scale

A session is one WebSocket and one pipeline, and nearly all of its time is spent waiting on the network — STT, LLM and TTS are all remote. It is I/O bound, not CPU bound, so concurrency per pod is high and the design is boring on purpose:

  • Stateless workers. Session context lives in Redis under calls:media:{session_id}, already published at open(). Any pod can serve any session, so the carrier's connection is load-balanced normally and no sticky routing is needed. The per-session bearer token authorises, not where the connection landed.
  • Bounded per pod. A worker accepts up to MEDIA_MAX_SESSIONS and refuses beyond it, so a saturated pod degrades predictably and the next connection goes elsewhere — instead of every call on that pod getting slower together.
  • Scaled on sessions, not CPU. CPU stays low while every call waits on a model, so a CPU-based autoscaler would not fire until the pod was already thrashing.
  • Reaped, not leaked. heartbeat exists and stale() already reaps: a worker that dies mid-call leaves a session that stops beating, and the call is finalised rather than sitting open forever.
  • The expensive part is not ours. Per-minute cost is the provider's STT and TTS, which is why the profile is per tenant — usage is attributable, and a tenant can move to a cheaper model without anyone redeploying.

Getting the socket into the cluster

The worker is useless if the upgrade never reaches it, and this is the part that looks fine right up until it doesn't: /media/health over plain HTTP answers correctly while the WebSocket 404s, because a proxy that drops Upgrade and Connection turns the handshake into an ordinary GET and the route stops matching.

The tracked answer is the commstate-media Ingress. It routes api.autocom.wexron.io/media(/|$)(.*) straight to the worker, rewriting to the /v1/calls/{session}/{token} the platform minted, and carries hour-long timeouts because a caller can be silent for minutes and the worker's proof of life is on the control channel rather than the socket. It is a separate Ingress rather than another path on commstate-public because ingress-nginx annotations are per-object, and sharing one would give every API request an hour-long read timeout.

It routes to the worker rather than through the in-cluster nginx: every audio frame crosses that hop, so the extra proxy is latency the caller hears.

cluster public entry status
live ingress-nginx (LoadBalancer on :80/:443) works as-is
UAT host nginx → NodePort 30350 → in-cluster nginx needs the host vhost to forward Upgrade/Connection

On UAT the Ingress is dormant, exactly as the rest of ingress.yaml already is — ingress-nginx's NodePorts are firewalled from the internet, so nothing reaches it until host nginx forwards to :30443. Until that swap the working route is the /media/ location in the in-cluster nginx ConfigMap, and the host vhost must carry the upgrade headers. Verified through ingress-nginx directly on the node: HTTP/1.1 101 Switching Protocols.

Phase 4 — AI to human, both in the browser

Already designed and largely built. The AI decides it cannot help, calls calls.transferToAgent, the platform picks an agent and redirects the Plivo leg to <Dial><User> their browser endpoint. The transcript is already carried across, so the agent's screen opens with the conversation on it.

What Phase 1 adds is that the human is now a browser tab rather than a mobile, which the transfer path does not need to know about.

What has to change in the model

One honest wrinkle. Call assumes a carrier: it has legs, a provider, and webhooks. A browser-to-AI session with no PSTN participant has none of those, and CallProviderContract requires originate, hangup and verifyWebhook — half of which a WebRTC-only session cannot implement.

That does not affect Phases 1–4, because in all of them the customer is on the PSTN and Plivo is the provider. It only matters if internal users should be able to hold a voice conversation with the AI from the app with no phone call involved — the existing push-to-talk assistant, made real-time.

If that is wanted, the clean answer is a separate provider = 'internal' with a narrower contract, not a pseudo-adapter that throws on half its methods. Worth deciding deliberately rather than discovering it in an adapter.

Configuration this adds

variable for notes
CALLS_MEDIA_WS_URL Phase 3 already read; AI calls apologise while unset
PLIVO_ENDPOINT_APP_ID Phase 1 application endpoints register against
OPENAI_API_KEY / GEMINI_API_KEY Phase 3 realtime model
DEEPGRAM_API_KEY etc. Phase 3 only for the composed pipeline

All are unset today, so nothing can do STT or TTS regardless of framework.

How the media plane actually behaves

The plan above is what was built. These are the things it turned out to depend on, each found by a call that failed.

A stream must never be the last verb

Plivo holds the carrier on <Stream keepCallAlive="true"> until the socket closes, then moves to the next verb. With no next verb the document ends and the call is dropped — so every ended stream was a dropped call: a worker restart, a crash, or a handover alike. The AI plan therefore ends with a redirect:

<Response>
  <Stream bidirectional="true" keepCallAlive="true">wss://…</Stream>
  <Redirect>https://…/webhook/calls/plivo/media-ended?call=…</Redirect>
</Response>

A handover works by ending the stream, not by the Transfer API

The Transfer API does not move a carrier parked on a stream. It answers call transferred and does nothing — so a handover looked complete in the logs while the assistant kept talking to a caller who had asked for a person. Not one redirect issued that way ever arrived.

The sequence that does work:

  1. the platform records where the caller is going (agent assigned, mode set),
  2. the worker closes its socket,
  3. the carrier follows the trailing redirect into media-ended,
  4. media-ended dials the already-assigned agent.

Step 4 must not re-decide or re-reserve. Re-deciding consults the number's configuration, which says "answer with the assistant", and hands the caller straight back. Re-reserving finds the agent at capacity — against this very call — and drops the caller into voicemail.

Transcription needs to be told what it is listening to

Two settings do most of the work on an 8 kHz line:

  • language — without it the transcriber detects per utterance, and an English caller came back as من عيد الله.
  • context_bias — up to 100 terms. Domain words have no acoustic margin, and they are the ones carrying the meaning: "transfer me to a human agent" came back as "transform into a human agent" and once as "Transcendence, nature, human nature". Add your product names and SKUs here.

The VAD also needs telephony settings; Pipecat's defaults are tuned for a headset and split sentences into fragments ("I am.", "available.") that the model answers with "I do not understand".

Silence during a tool call reads as a dropped line

A lookup takes seconds. The worker speaks a short rotating filler while one runs, because callers talk over silence or hang up before the answer arrives.

Nothing transient is left to a webhook that may not come

ringing and transferring are states a call passes through. When the callback that should move it never arrives, a reaper finishes the call and gives back the agent it was holding — a reservation that outlives its call is invisible, and shows up only as "nobody available" on every later handover.

The media control channel authenticates with a session token, which identifies a call and not a tenant. Its routes run inside the call's tenant via middleware — without it the handover's "first active queue" fallback picks the lowest-numbered queue on the *whole platform*, and the tool calls read another tenant's catalogue.

What this plan deliberately avoids

  • A second real-time system. Plivo already terminates both legs; adding LiveKit for the agent side would mean bridging two media domains on every transferred call.
  • Rewriting the turn-based assistant. It works, it is shipped, and it stays until the streaming path is proven.
  • Streaming STT of our own. The existing providers are file-based by design. Pipecat's integrations handle streaming; reusing the file-based ones for real-time would be the wrong tool.