Kubernetes

The chart in turncall-charts (charts/turncall; commands below run from a checkout of it) runs TurnCall against a Postgres (with pgvector) and a Redis you already have — or, for a laptop, brings its own — and, if you want it, the builder beside it. It is sized for the one thing a voice platform cannot get wrong on Kubernetes: a pod that is scaled down or replaced keeps its calls until they end. See Graceful shutdown for how a process drains. There are two ways to install it — the same chart with different values:

	Install (production)	Everything in the cluster (development)
For	a real cluster	a laptop: Docker Desktop, kind, minikube
Postgres and Redis	yours, by URL	run by the chart, one pod each
Files	S3	a volume
Secrets	a Secret you create	dev values, plus your keys file
Images	the released image, pinned	local `:dev` builds
Builder	off unless enabled	on
API pods	2 or more, optionally autoscaled	1
Reached through	an Ingress with TLS	`kubectl port-forward` (and ngrok for phone calls)

The production path assumes you run the infrastructure and adds TurnCall to it; the development path is the docker compose stack on Kubernetes — convenient, with no backups or replication.

What it runs

Deployment	Process	Replicas
`<release>-api`	the API, Twilio/Telnyx/WhatsApp webhooks and media streams, WebRTC	`api.replicas`, or the autoscaler
`<release>-eval-worker`	`turncall-eval-worker`	`evalWorker.replicas`
`<release>-campaign-worker`	`turncall-campaign-worker`	`campaignWorker.replicas`
`<release>-asterisk-edge`	the API process holding Asterisk’s ARI connection	exactly 1, when `asteriskEdge.enabled`
`<release>-postgresql`, `<release>-redis` (StatefulSets)	Postgres with pgvector, Redis	1 each, when `postgresql.enabled` / `redis.enabled`
`<release>-builder-api`, `<release>-builder-web`	the builder’s API and its console	1 each, when `builder.enabled`

The workers are the same image with another entrypoint and never run inside the API: their work in the API’s event loop would become dead air on a live call. Migrations (alembic upgrade head) run as a Job before any pod of a new version starts, on install and on every upgrade — or, with the bundled Postgres, in the API pod (see below).

Install (production)

TurnCall against your own Postgres, Redis and S3, with secrets in a Secret you manage.

kubectl create secret generic turncall-env \
  --from-literal=DATABASE_URL='postgresql+asyncpg://turncall:...@postgres:5432/turncall' \
  --from-literal=REDIS_URL='redis://redis:6379/0' \
  --from-literal=PLATFORM_API_KEY='...' \
  --from-literal=API_KEY_HASH_SECRET='...' \
  --from-literal=CREDENTIAL_ENCRYPTION_KEY='...' \
  --from-literal=TWILIO_ACCOUNT_SID='...' --from-literal=TWILIO_AUTH_TOKEN='...' \
  --from-literal=OPENAI_API_KEY='...' --from-literal=DEEPGRAM_API_KEY='...'

helm install turncall ./charts/turncall \
  --set existingSecret=turncall-env \
  --set image.tag=<release> \
  --set config.S3_BUCKET_NAME=turncall-recordings \
  --set config.PUBLIC_BASE_URL=https://turncall.example.com

values-production.yaml is the same install as a file to copy: a pinned image, existingSecret, S3, publicBaseUrl, and an Ingress with TLS and the WebSocket timeouts — plus the builder beside it; set builder.enabled=false for TurnCall alone. Replace its example.com hosts and placeholders. existingSecret holds every secret setting as an environment variable; config holds the rest. Both take the variables listed in the project’s environment reference. secretEnv is the alternative to existingSecret when you would rather have the chart own the Secret.

Pin image.tag to a release. A release without the drain would drop calls on every scale-down this chart performs. Every merge to main also publishes ghcr.io/kobikis/turncall:main and :sha-<commit>; pin a sha- tag to run code newer than the last release.
Use S3 storage (STORAGE_BACKEND: s3, the chart’s default). local storage is one pod’s filesystem, so a recording written on one pod is missing on the next.
Expose the API with ingress.enabled or your own route to the <release> Service. The chart’s Ingress routes /v1, /webhooks and /ws only (ingress.paths): /metrics, /ready, /live and /health carry no credential and report how many calls a pod is carrying, so they stay inside the cluster, where the kubelet and the PodMonitor reach each pod directly. A route of your own should leave them out the same way. Media streams are WebSockets that last a whole call, so raise your ingress controller’s proxy timeouts to at least your longest call (the annotations for ingress-nginx are in values.yaml).

Everything in the cluster (development)

values-local.yaml is the docker compose stack on Kubernetes (Docker Desktop, kind, minikube): the chart also runs Postgres (pgvector/pgvector:pg16) and Redis (redis:7-alpine), one pod and one volume each, keeps files on a volume (STORAGE_BACKEND: local), and fills DATABASE_URL and REDIS_URL itself. It also runs the builder; add --set builder.enabled=false for TurnCall alone. It runs local image tags, which it expects to find in your Docker — the released TurnCall image is linux/amd64 only. Build them from the code repos, or point each image at your registry’s:

docker build -t turncall:dev turncall
docker build -t turncall-builder-api:dev turncall-builder-api
docker build -t turncall-agent-runtime:dev turncall-builder-api/agent-runtime
docker build -t turncall-builder-web:dev turncall-builder-web

Phone calls need a public https URL that reaches the API, so start the tunnel first and copy the URL it prints. On a free ngrok account it is a fixed domain, such as https://<name>.ngrok-free.dev, so it survives a restart:

ngrok http 8090

Then the keys file, with that URL as publicBaseUrl:

cat > my-keys.yaml <<'YAML'
publicBaseUrl: https://<name>.ngrok-free.dev   # ngrok's URL; Twilio/Telnyx call it
secretEnv:
  OPENAI_API_KEY: sk-...
  DEEPGRAM_API_KEY: ...
  TWILIO_ACCOUNT_SID: AC...
  TWILIO_AUTH_TOKEN: ...
builder:
  secretEnv:
    ANTHROPIC_API_KEY: sk-ant-...   # or OPENAI_API_KEY: the model the builder talks with
YAML

helm install turncall ./charts/turncall -n turncall --create-namespace \
  -f charts/turncall/values-local.yaml -f my-keys.yaml
kubectl -n turncall port-forward svc/turncall 8090:80

The API is then on localhost:8090 — and at the ngrok URL, which forwards to the port-forward — with the dev PLATFORM_API_KEY from values-local.yaml for creating a project. publicBaseUrl becomes TurnCall’s PUBLIC_BASE_URL (config.PUBLIC_BASE_URL wins if both are set) and the URL the builder’s console shows as a number’s webhook. Without it, a number bound is saved but not configured at the carrier, and the console says so. If the URL changes, or you installed without it:

helm upgrade turncall ./charts/turncall -n turncall --reuse-values --set publicBaseUrl=https://<name>.ngrok-free.dev

Then re-save each number in the console, so TurnCall points it at the new URL.

Migrations run in the API pod’s migrate init container, retrying until Postgres is up: the hook Job runs before install creates anything, so it cannot migrate a database the same install brings. The workers wait in a wait-for-migrations init container until the database is at the image’s head revision, so they start against the tables they expect — on upgrades too.
One API pod. The files volume is ReadWriteOnce.
A rebuilt image under the same tag is picked up by kubectl rollout restart deploy: values-local.yaml sets pullPolicy: Always, and Docker Desktop’s cluster pulls local tags from the Docker engine. On kind or minikube, where images are loaded onto the nodes, set image.pullPolicy=IfNotPresent.
Volumes outlive helm uninstall — Postgres’s and Redis’s (a StatefulSet’s claims), the files one and the builder’s generated repos. kubectl delete namespace turncall starts over.
After pulling a newer chart, upgrade with --reset-then-reuse-values: the chart’s new defaults plus the values you installed with. --reuse-values alone keeps the old defaults too.
Not for production: no replication, no backups, dev secrets. Each piece is a switch (postgresql.enabled, redis.enabled, localStorage.enabled), so moving to managed services is turning them off and setting the URLs.

The builder

builder.enabled adds the agent builder — builder-api and its console — to the same release. Off by default: TurnCall installs alone.

It finds TurnCall itself: the builder talks to this release’s TurnCall Service and reads TurnCall’s PLATFORM_API_KEY from TurnCall’s Secret, so the two can never disagree. publicBaseUrl is also what the console shows as a number’s webhook.
Its database is builder on the bundled Postgres, created on first start, or builder.secretEnv.DATABASE_URL (or builder.existingSecret) for your own.
Each agent’s backend — its tool webhooks, call-init and event feed — runs as a Deployment named for the agent and a Service, which builder-api creates through the Kubernetes API, holding a Role over just those kinds of object. All of them run one runtime image (builder.agentRuntime.image) with the agent’s code mounted, so nothing is built in the cluster. TurnCall reaches them at http://turncall-agent-<port>.<namespace>.svc.cluster.local:8000, which is why a production WebRTC server_url may be an in-cluster Service.
Agent Backends’ namespace (builder.agentNamespace, default the release’s): in production, give them one of their own. They run code the builder generated, and builder-api’s Role — which covers Secrets — is then granted there instead of beside TurnCall’s Secret.
The console is the only part a browser needs: it serves the UI and proxies /api to builder-api, so Google login and Pipedream callbacks come back through it (https://<console>/api/...). Expose it with builder.ingress; a builder turn streams for minutes, so turn the controller’s buffering off and raise its read timeout.
Private images: imagePullSecrets reaches every pod, Agent Backends included.

On a laptop, log in with a dev account:

kubectl -n turncall exec deploy/turncall-builder-api -- python -m scripts.seed_guest
kubectl -n turncall port-forward svc/turncall-builder-web 5173:80   # http://localhost:5173, guest@turncall.local / guest

seed_guest refuses when builder.config.TURNCALL_ENV is production.

Grace periods

Kubernetes sends SIGKILL when a pod’s terminationGracePeriodSeconds runs out, and a killed pod drops its calls. The chart sizes each grace period from the work that process may still be holding:

Deployment	Grace period	Why
API	`api.preStopSeconds` + `maxCallDurationSeconds` + 90	The drain waits for the longest call, plus 30s to build and finalize it and 60s for post-call webhooks
eval worker	max(`evals.maxRunDurationSeconds`, 180 × `evals.maxIterations`) + 60	It finishes the runs it holds; that is the longest one may take. A run killed sooner is marked `errored` by the janitor, not lost
campaign worker	`campaignWorker.terminationGracePeriodSeconds` (60)	It stops between passes, so a dial is recorded before it exits. A worker killed mid-dial leaves the target’s outcome unknown, and it is never redialled
Asterisk edge	90	It carries no media, so no calls of its own to wait for

Set maxCallDurationSeconds to the longest max_call_duration_seconds among your agents (default 3600). It is the one value the chart cannot read for you. Lower it on agents that never need an hour and every scale-down gets shorter. The API pods also use:

a preStop sleep (api.preStopSeconds, 10), during which the terminating pod leaves the Service’s endpoints and load balancers catch up;
readiness on /ready, which fails from the moment the drain starts;
liveness and startup on /live, which stays green while draining;
maxUnavailable: 0 on rollouts, so a new pod is ready before an old one starts to drain.

Asterisk

Asterisk’s ARI hands an app’s calls to the newest connection only, and the connection it replaced never gets them back. Two processes holding it would split calls between a process that knows them and one that does not. So the chart never lets an API replica hold it: every API pod runs with ASTERISK_EDGE_ENABLED=false, and asteriskEdge.enabled: true adds one Deployment that holds it:

one replica, never autoscaled;
Recreate on rollout, so the old edge is gone before the new one connects;
not behind the Service. Each call’s media stream reaches the API pods through the websocket_client.conf URI you gave Asterisk.

Enable it if the platform or any project uses Asterisk. With it off, nothing connects to Asterisk. Restarting the edge has a cost, so change it in a quiet hour:

For the seconds between the old pod exiting and the new one connecting, Asterisk has no turncall app, and Stasis() returns with STASISSTATUS=FAILED. Route that to your fallback in the dialplan.
Calls already connected keep their agent, but the new edge does not know them. When one of those callers hangs up, nothing ends the agent’s side, which runs until its idle guard (or max_call_duration_seconds) ends it.

Autoscaling

Scale the API on live calls per pod, not CPU. A pod runs every call’s audio on one event loop, so its limit is set by audio underruns and turn latency, not by average CPU. The chart’s HorizontalPodAutoscaler targets an average of autoscaling.activeCallsPerPod across the API pods, read from the turncall_active_calls gauge on each pod’s /metrics. The chart’s default target is 2 calls per pod, with requests.cpu: 1, requests.memory: 1Gi and no CPU limit. That is the measured capacity for the default cascade agent: 2 concurrent calls with no audio gaps on one dedicated core, the first gaps at 4. It was measured on an Apple M1 Pro, in a Docker Desktop VM with 8 vCPU, on a busy host, so treat it as a floor and re-measure on your own node type before raising it.

Do not set a CPU limit. A CPU quota freezes the whole process for the rest of a 100 ms period whenever it bursts past it, and since every call’s audio runs on one event loop, each freeze is a gap on every call at once. Under limits.cpu: 1 a pod carried only 1 call cleanly; a second CPU of quota did not help, because the loop is single-threaded.
Memory is about 265 MiB once the first call has loaded Smart Turn, plus 45 MiB per call. The memory limit (2Gi) sits well above the request: reaching it is an OOM kill, which drops every call on the pod.

Scale-down is deliberately slow (one pod per 5 minutes, after 10 minutes of low load): a removed pod drains its calls before it goes, so quick scale-downs churn pods that are only draining. Kubernetes picks which pod to remove without knowing how many calls each carries.

With prometheus-adapter

The HPA reads turncall_active_calls from the custom metrics API. With Prometheus scraping each API pod (podMonitor.enabled for the Prometheus operator, or the prometheus.io/* annotations the pods carry), give prometheus-adapter this rule:

rules:
  custom:
    - seriesQuery: 'turncall_active_calls{namespace!="",pod!=""}'
      resources:
        overrides:
          namespace: {resource: namespace}
          pod: {resource: pod}
      name: {as: turncall_active_calls}
      metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)'

Then:

helm upgrade turncall ./charts/turncall --reuse-values --set autoscaling.enabled=true
kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/turncall_active_calls"

Scrape each pod, never the Service: the gauge is per process, and a load-balanced scrape reads a random pod’s count.

With KEDA

Leave autoscaling.enabled off and give KEDA the same target:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: turncall-api
spec:
  scaleTargetRef:
    name: turncall-api            # <release>-api
  minReplicaCount: 2
  maxReplicaCount: 10
  advanced:
    horizontalPodAutoscalerConfig:
      behavior:
        scaleDown:
          stabilizationWindowSeconds: 600
          policies: [{type: Pods, value: 1, periodSeconds: 300}]
  triggers:
    - type: prometheus
      metadata:
        serverAddress: http://prometheus.monitoring:9090
        query: sum(turncall_active_calls{namespace="default", pod=~"turncall-api-.*"})
        threshold: "<autoscaling.activeCallsPerPod>"

KEDA divides the query’s total by the threshold to get a replica count, which is the same arithmetic as the HPA’s average.

Not covered

WebRTC media is UDP between the browser and the pod, which an Ingress does not carry. Browser calls — the builder’s test calls included — need the pods reachable over UDP (or a TURN server), which is outside this chart. Phone calls are unaffected: their audio arrives over the carrier’s WebSocket.
Postgres and Redis in production are yours to run (the bundled ones are for development). Postgres needs the vector extension available; the first migration creates it.

Getting Started

Guides

What it runs

Install (production)

Everything in the cluster (development)

The builder

Grace periods

Asterisk

Autoscaling

With prometheus-adapter

With KEDA

Not covered

​What it runs

​Install (production)

​Everything in the cluster (development)

​The builder

​Grace periods

​Asterisk

​Autoscaling

​With prometheus-adapter

​With KEDA

​Not covered

What it runs

Install (production)

Everything in the cluster (development)

The builder

Grace periods

Asterisk

Autoscaling

With prometheus-adapter

With KEDA

Not covered