Speech-to-Speech

Speech-to-Speech (S2S) mode skips separate STT and TTS stages — the model handles audio natively for ultra-low latency (~300ms).

Configuration

Set pipeline_mode: "s2s" in the agent config. The stt, llm, and tts fields are ignored in S2S mode.

{
  "pipeline_mode": "s2s",
  "s2s": {
    "provider": "openai",
    "model": "gpt-4o-realtime-preview",
    "voice": "alloy"
  }
}

{
  "pipeline_mode": "s2s",
  "s2s": {
    "provider": "google",
    "model": "models/gemini-3.1-flash-live-preview",
    "voice": "Kore"
  }
}

{
  "pipeline_mode": "s2s",
  "s2s": {
    "provider": "openai",
    "base_url": "wss://ai-gateway.vercel.sh/v1/realtime",
    "model": "xai/grok-voice-think-fast-1.0",
    "voice": "cosmo"
  }
}

Options

Field	Required	Description
`provider`	Yes	`openai` or `google`
`model`	No	Model name (defaults per provider). Gemini default: `models/gemini-3.1-flash-live-preview`
`voice`	No	Voice name (default: `alloy` / `Charon`)
`turn_detection`	No	`server_vad` (default) or `pipecat_vad`
`base_url`	No	`wss://` endpoint of an OpenAI-Realtime-compatible gateway (`openai` provider only). See Gateways

Voices

OpenAI Realtime — one of a fixed set: alloy, ash, ballad, coral, echo, sage, shimmer, verse. An unknown voice is rejected on create.
Gemini Live — any Gemini-supported voice (e.g. Aoede, Charon, Kore, Puck, Zephyr, and more). The set grows per model, so TurnCall doesn’t allowlist it — Gemini validates the voice when the session connects.

Gateways and third-party models

The openai provider speaks the OpenAI-Realtime WebSocket protocol — and so do OpenAI-Realtime-compatible gateways (Vercel AI Gateway, LiteLLM) and xAI direct. Set s2s.base_url to route the realtime connection through one; provider-prefixed models like xai/grok-voice-think-fast-1.0 or openai/gpt-realtime-2 then flow over the same protocol — no new provider, same pipeline. Two requirements:

Allowlist the gateway URL

base_url is an outbound target, so it’s gated by the same SSRF guard as custom LLM endpoints. Add its wss:// pattern to BYOM_ALLOWED_URL_PATTERNS, e.g. ["wss://ai-gateway.vercel.sh/*"]. A base_url outside the allowlist is rejected before the call starts.

Use the gateway key

Set OPENAI_API_KEY to the gateway’s API key — it’s sent as Authorization: Bearer … on the WebSocket.

When base_url is set, the OpenAI voice allowlist is bypassed — the gateway routes to models with their own voice sets (e.g. Grok), which it validates upstream. Pass the target model’s own voice name.

Pipeline

transport.input → user_agg (VAD)
  → S2S_LLM (OpenAI Realtime / Gemini Live WebSocket)
  → transport.output
  → context_aggregator.assistant → observability

Required API Keys

Provider	Environment Variable
OpenAI Realtime	`OPENAI_API_KEY`
Gemini Live	`GOOGLE_API_KEY`
Gateway (Vercel AI Gateway / LiteLLM / xAI)	`OPENAI_API_KEY` set to the gateway key

S2S mode cannot be combined with voicemail_detection.enabled: true.

Getting Started

Guides

Configuration

Options

Voices

Gateways and third-party models

Pipeline

Required API Keys

​Configuration

​Options

​Voices

​Gateways and third-party models

​Pipeline

​Required API Keys

Configuration

Options

Voices

Gateways and third-party models

Pipeline

Required API Keys