> ## Documentation Index > Fetch the complete documentation index at: https://docs.turncall.io/llms.txt > Use this file to discover all available pages before exploring further. # Kubernetes > The Helm chart: the API, the workers, migrations, autoscaling on live calls, and the builder beside them The chart in [turncall-charts](https://github.com/kobikis/turncall-charts) (`charts/turncall`; commands below run from a checkout of it) runs TurnCall against a Postgres (with `pgvector`) and a Redis you already have — or, for a laptop, [brings its own](#everything-in-the-cluster-development) — and, if you want it, [the builder](#the-builder) beside it. It is sized for the one thing a voice platform cannot get wrong on Kubernetes: a pod that is scaled down or replaced keeps its calls until they end. See [Graceful shutdown](/guides/graceful-shutdown) for how a process drains. There are two ways to install it — the same chart with different values: | | [Install (production)](#install-production) | [Everything in the cluster (development)](#everything-in-the-cluster-development) | | - | - | - | | For | a real cluster | a laptop: Docker Desktop, kind, minikube | | Postgres and Redis | yours, by URL | run by the chart, one pod each | | Files | S3 | a volume | | Secrets | a Secret you create | dev values, plus your keys file | | Images | the released image, pinned | local `:dev` builds | | Builder | off unless enabled | on | | API pods | 2 or more, optionally autoscaled | 1 | | Reached through | an Ingress with TLS | `kubectl port-forward` (and ngrok for phone calls) | The production path assumes you run the infrastructure and adds TurnCall to it; the development path is the docker compose stack on Kubernetes — convenient, with no backups or replication. ## What it runs | Deployment | Process | Replicas | | - | - | - | | `-api` | the API, Twilio/Telnyx/WhatsApp webhooks and media streams, WebRTC | `api.replicas`, or the autoscaler | | `-eval-worker` | `turncall-eval-worker` | `evalWorker.replicas` | | `-campaign-worker` | `turncall-campaign-worker` | `campaignWorker.replicas` | | `-asterisk-edge` | the API process holding Asterisk's ARI connection | exactly 1, when `asteriskEdge.enabled` | | `-postgresql`, `-redis` (StatefulSets) | Postgres with pgvector, Redis | 1 each, when `postgresql.enabled` / `redis.enabled` | | `-builder-api`, `-builder-web` | the builder's API and its console | 1 each, when `builder.enabled` | The workers are the same image with another entrypoint and never run inside the API: their work in the API's event loop would become dead air on a live call. Migrations (`alembic upgrade head`) run as a Job before any pod of a new version starts, on install and on every upgrade — or, with the bundled Postgres, in the API pod (see [below](#everything-in-the-cluster-development)). ## Install (production) TurnCall against your own Postgres, Redis and S3, with secrets in a Secret you manage. ```bash theme={null} kubectl create secret generic turncall-env \ --from-literal=DATABASE_URL='postgresql+asyncpg://turncall:...@postgres:5432/turncall' \ --from-literal=REDIS_URL='redis://redis:6379/0' \ --from-literal=PLATFORM_API_KEY='...' \ --from-literal=API_KEY_HASH_SECRET='...' \ --from-literal=CREDENTIAL_ENCRYPTION_KEY='...' \ --from-literal=TWILIO_ACCOUNT_SID='...' --from-literal=TWILIO_AUTH_TOKEN='...' \ --from-literal=OPENAI_API_KEY='...' --from-literal=DEEPGRAM_API_KEY='...' helm install turncall ./charts/turncall \ --set existingSecret=turncall-env \ --set image.tag= \ --set config.S3_BUCKET_NAME=turncall-recordings \ --set config.PUBLIC_BASE_URL=https://turncall.example.com ``` [`values-production.yaml`](https://github.com/kobikis/turncall-charts/tree/main/charts/turncall/values-production.yaml) is the same install as a file to copy: a pinned image, `existingSecret`, S3, `publicBaseUrl`, and an Ingress with TLS and the WebSocket timeouts — plus the builder beside it; set `builder.enabled=false` for TurnCall alone. Replace its `example.com` hosts and placeholders. `existingSecret` holds every secret setting as an environment variable; `config` holds the rest. Both take the variables listed in the project's environment reference. `secretEnv` is the alternative to `existingSecret` when you would rather have the chart own the Secret. * **Pin `image.tag`** to a release. A release without the drain would drop calls on every scale-down this chart performs. Every merge to `main` also publishes `ghcr.io/kobikis/turncall:main` and `:sha-`; pin a `sha-` tag to run code newer than the last release. * **Use S3 storage** (`STORAGE_BACKEND: s3`, the chart's default). `local` storage is one pod's filesystem, so a recording written on one pod is missing on the next. * **Expose the API** with `ingress.enabled` or your own route to the `` Service. The chart's Ingress routes `/v1`, `/webhooks` and `/ws` only (`ingress.paths`): `/metrics`, `/ready`, `/live` and `/health` carry no credential and report how many calls a pod is carrying, so they stay inside the cluster, where the kubelet and the PodMonitor reach each pod directly. A route of your own should leave them out the same way. Media streams are WebSockets that last a whole call, so raise your ingress controller's proxy timeouts to at least your longest call (the annotations for ingress-nginx are in `values.yaml`). ## Everything in the cluster (development) `values-local.yaml` is the docker compose stack on Kubernetes (Docker Desktop, kind, minikube): the chart also runs Postgres (`pgvector/pgvector:pg16`) and Redis (`redis:7-alpine`), one pod and one volume each, keeps files on a volume (`STORAGE_BACKEND: local`), and fills `DATABASE_URL` and `REDIS_URL` itself. It also runs [the builder](#the-builder); add `--set builder.enabled=false` for TurnCall alone. It runs local image tags, which it expects to find in your Docker — the released TurnCall image is linux/amd64 only. Build them from the code repos, or point each `image` at your registry's: ```bash theme={null} docker build -t turncall:dev turncall docker build -t turncall-builder-api:dev turncall-builder-api docker build -t turncall-agent-runtime:dev turncall-builder-api/agent-runtime docker build -t turncall-builder-web:dev turncall-builder-web ``` Phone calls need a public https URL that reaches the API, so start the tunnel first and copy the URL it prints. On a free ngrok account it is a fixed domain, such as `https://.ngrok-free.dev`, so it survives a restart: ```bash theme={null} ngrok http 8090 ``` Then the keys file, with that URL as `publicBaseUrl`: ```bash theme={null} cat > my-keys.yaml <<'YAML' publicBaseUrl: https://.ngrok-free.dev # ngrok's URL; Twilio/Telnyx call it secretEnv: OPENAI_API_KEY: sk-... DEEPGRAM_API_KEY: ... TWILIO_ACCOUNT_SID: AC... TWILIO_AUTH_TOKEN: ... builder: secretEnv: ANTHROPIC_API_KEY: sk-ant-... # or OPENAI_API_KEY: the model the builder talks with YAML helm install turncall ./charts/turncall -n turncall --create-namespace \ -f charts/turncall/values-local.yaml -f my-keys.yaml kubectl -n turncall port-forward svc/turncall 8090:80 ``` The API is then on `localhost:8090` — and at the ngrok URL, which forwards to the port-forward — with the dev `PLATFORM_API_KEY` from `values-local.yaml` for creating a project. `publicBaseUrl` becomes TurnCall's `PUBLIC_BASE_URL` (`config.PUBLIC_BASE_URL` wins if both are set) and the URL the builder's console shows as a number's webhook. Without it, a number bound is saved but not configured at the carrier, and the console says so. If the URL changes, or you installed without it: ```bash theme={null} helm upgrade turncall ./charts/turncall -n turncall --reuse-values --set publicBaseUrl=https://.ngrok-free.dev ``` Then re-save each number in the console, so TurnCall points it at the new URL. * **Migrations** run in the API pod's `migrate` init container, retrying until Postgres is up: the hook Job runs before install creates anything, so it cannot migrate a database the same install brings. The workers wait in a `wait-for-migrations` init container until the database is at the image's head revision, so they start against the tables they expect — on upgrades too. * **One API pod.** The files volume is `ReadWriteOnce`. * **A rebuilt image** under the same tag is picked up by `kubectl rollout restart deploy`: `values-local.yaml` sets `pullPolicy: Always`, and Docker Desktop's cluster pulls local tags from the Docker engine. On kind or minikube, where images are loaded onto the nodes, set `image.pullPolicy=IfNotPresent`. * **Volumes outlive `helm uninstall`** — Postgres's and Redis's (a StatefulSet's claims), the files one and the builder's generated repos. `kubectl delete namespace turncall` starts over. * **After pulling a newer chart**, upgrade with `--reset-then-reuse-values`: the chart's new defaults plus the values you installed with. `--reuse-values` alone keeps the old defaults too. * **Not for production**: no replication, no backups, dev secrets. Each piece is a switch (`postgresql.enabled`, `redis.enabled`, `localStorage.enabled`), so moving to managed services is turning them off and setting the URLs. ## The builder `builder.enabled` adds the agent builder — builder-api and its console — to the same release. Off by default: TurnCall installs alone. * **It finds TurnCall itself**: the builder talks to this release's TurnCall Service and reads TurnCall's `PLATFORM_API_KEY` from TurnCall's Secret, so the two can never disagree. `publicBaseUrl` is also what the console shows as a number's webhook. * **Its database** is `builder` on the bundled Postgres, created on first start, or `builder.secretEnv.DATABASE_URL` (or `builder.existingSecret`) for your own. * **Each agent's backend** — its tool webhooks, call-init and event feed — runs as a Deployment named for the agent and a Service, which builder-api creates through the Kubernetes API, holding a Role over just those kinds of object. All of them run one runtime image (`builder.agentRuntime.image`) with the agent's code mounted, so nothing is built in the cluster. TurnCall reaches them at `http://turncall-agent-..svc.cluster.local:8000`, which is why a production WebRTC `server_url` may be an in-cluster Service. * **Agent Backends' namespace** (`builder.agentNamespace`, default the release's): in production, give them one of their own. They run code the builder generated, and builder-api's Role — which covers Secrets — is then granted there instead of beside TurnCall's Secret. * **The console** is the only part a browser needs: it serves the UI and proxies `/api` to builder-api, so Google login and Pipedream callbacks come back through it (`https:///api/...`). Expose it with `builder.ingress`; a builder turn streams for minutes, so turn the controller's buffering off and raise its read timeout. * **Private images**: `imagePullSecrets` reaches every pod, Agent Backends included. On a laptop, log in with a dev account: ```bash theme={null} kubectl -n turncall exec deploy/turncall-builder-api -- python -m scripts.seed_guest kubectl -n turncall port-forward svc/turncall-builder-web 5173:80 # http://localhost:5173, guest@turncall.local / guest ``` `seed_guest` refuses when `builder.config.TURNCALL_ENV` is `production`. ## Grace periods Kubernetes sends `SIGKILL` when a pod's `terminationGracePeriodSeconds` runs out, and a killed pod drops its calls. The chart sizes each grace period from the work that process may still be holding: | Deployment | Grace period | Why | | - | - | - | | API | `api.preStopSeconds` + `maxCallDurationSeconds` + 90 | The drain waits for the longest call, plus 30s to build and finalize it and 60s for post-call webhooks | | eval worker | max(`evals.maxRunDurationSeconds`, 180 × `evals.maxIterations`) + 60 | It finishes the runs it holds; that is the longest one may take. A run killed sooner is marked `errored` by the janitor, not lost | | campaign worker | `campaignWorker.terminationGracePeriodSeconds` (60) | It stops between passes, so a dial is recorded before it exits. A worker killed mid-dial leaves the target's outcome unknown, and it is never redialled | | Asterisk edge | 90 | It carries no media, so no calls of its own to wait for | **Set `maxCallDurationSeconds` to the longest `max_call_duration_seconds` among your agents** (default `3600`). It is the one value the chart cannot read for you. Lower it on agents that never need an hour and every scale-down gets shorter. The API pods also use: * a `preStop` sleep (`api.preStopSeconds`, 10), during which the terminating pod leaves the Service's endpoints and load balancers catch up; * readiness on `/ready`, which fails from the moment the drain starts; * liveness and startup on `/live`, which stays green while draining; * `maxUnavailable: 0` on rollouts, so a new pod is ready before an old one starts to drain. ## Asterisk Asterisk's ARI hands an app's calls to the newest connection only, and the connection it replaced never gets them back. Two processes holding it would split calls between a process that knows them and one that does not. So the chart never lets an API replica hold it: every API pod runs with `ASTERISK_EDGE_ENABLED=false`, and `asteriskEdge.enabled: true` adds one Deployment that holds it: * one replica, never autoscaled; * `Recreate` on rollout, so the old edge is gone before the new one connects; * not behind the Service. Each call's media stream reaches the API pods through the `websocket_client.conf` URI you gave Asterisk. Enable it if the platform or any project uses Asterisk. With it off, nothing connects to Asterisk. Restarting the edge has a cost, so change it in a quiet hour: * For the seconds between the old pod exiting and the new one connecting, Asterisk has no `turncall` app, and `Stasis()` returns with `STASISSTATUS=FAILED`. Route that to your fallback in the dialplan. * Calls already connected keep their agent, but the new edge does not know them. When one of those callers hangs up, nothing ends the agent's side, which runs until its idle guard (or `max_call_duration_seconds`) ends it. ## Autoscaling Scale the API on live calls per pod, not CPU. A pod runs every call's audio on one event loop, so its limit is set by audio underruns and turn latency, not by average CPU. The chart's `HorizontalPodAutoscaler` targets an average of `autoscaling.activeCallsPerPod` across the API pods, read from the `turncall_active_calls` gauge on each pod's `/metrics`. The chart's default target is **2 calls per pod**, with `requests.cpu: 1`, `requests.memory: 1Gi` and **no CPU limit**. That is the [measured capacity](/guides/capacity) for the default cascade agent: 2 concurrent calls with no audio gaps on one dedicated core, the first gaps at 4. It was measured on an Apple M1 Pro, in a Docker Desktop VM with 8 vCPU, on a busy host, so treat it as a floor and re-measure on your own node type before raising it. * **Do not set a CPU limit.** A CPU quota freezes the whole process for the rest of a 100 ms period whenever it bursts past it, and since every call's audio runs on one event loop, each freeze is a gap on every call at once. Under `limits.cpu: 1` a pod carried only 1 call cleanly; a second CPU of quota did not help, because the loop is single-threaded. * **Memory** is about 265 MiB once the first call has loaded Smart Turn, plus 45 MiB per call. The memory limit (2Gi) sits well above the request: reaching it is an OOM kill, which drops every call on the pod. Scale-down is deliberately slow (one pod per 5 minutes, after 10 minutes of low load): a removed pod drains its calls before it goes, so quick scale-downs churn pods that are only draining. Kubernetes picks which pod to remove without knowing how many calls each carries. ### With prometheus-adapter The HPA reads `turncall_active_calls` from the custom metrics API. With Prometheus scraping each API pod (`podMonitor.enabled` for the Prometheus operator, or the `prometheus.io/*` annotations the pods carry), give [prometheus-adapter](https://github.com/kubernetes-sigs/prometheus-adapter) this rule: ```yaml theme={null} rules: custom: - seriesQuery: 'turncall_active_calls{namespace!="",pod!=""}' resources: overrides: namespace: {resource: namespace} pod: {resource: pod} name: {as: turncall_active_calls} metricsQuery: 'sum(<<.Series>>{<<.LabelMatchers>>}) by (<<.GroupBy>>)' ``` Then: ```bash theme={null} helm upgrade turncall ./charts/turncall --reuse-values --set autoscaling.enabled=true kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/namespaces/default/pods/*/turncall_active_calls" ``` Scrape each pod, never the Service: the gauge is per process, and a load-balanced scrape reads a random pod's count. ### With KEDA Leave `autoscaling.enabled` off and give KEDA the same target: ```yaml theme={null} apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: turncall-api spec: scaleTargetRef: name: turncall-api # -api minReplicaCount: 2 maxReplicaCount: 10 advanced: horizontalPodAutoscalerConfig: behavior: scaleDown: stabilizationWindowSeconds: 600 policies: [{type: Pods, value: 1, periodSeconds: 300}] triggers: - type: prometheus metadata: serverAddress: http://prometheus.monitoring:9090 query: sum(turncall_active_calls{namespace="default", pod=~"turncall-api-.*"}) threshold: "" ``` KEDA divides the query's total by the threshold to get a replica count, which is the same arithmetic as the HPA's average. ## Not covered * **WebRTC** media is UDP between the browser and the pod, which an Ingress does not carry. Browser calls — the builder's test calls included — need the pods reachable over UDP (or a TURN server), which is outside this chart. Phone calls are unaffected: their audio arrives over the carrier's WebSocket. * **Postgres and Redis** in production are yours to run (the bundled ones are for development). Postgres needs the `vector` extension available; the first migration creates it. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.