The figure
For the default cascade agent (Silero VAD, Smart Turn V3, barge-in, idle guard, recording; providers elsewhere):
Per-pod target: 2 concurrent calls on
requests.cpu: 1, no CPU limit, 1 GiB
of memory. 2 is the largest N that stayed clean on a dedicated core; 4 is
where the first gaps appeared there (2 per minute of agent speech, ≤ 52 ms).
Under a CPU quota only 1 call stayed clean. The measuring
machine was noisy (below) and aarch64, so this is a floor: rerun make load-test on your node type before raising the target, and raise it to the
largest step that comes back ok.
Why those settings:
- No CPU limit. A CFS quota freezes the whole process for the rest of a 100 ms period whenever its threads (ONNX inference, the logging thread) burst past it, even at 30–45 % average CPU: 8–17 % of periods were throttled at 2–4 calls, and each throttle is a loop stall of up to ~100 ms. The same core without a quota held p95 turn latency flat to 8 calls (vs 2) and cut underruns 4–50x at every step. A second CPU of quota did not help either: the event loop is single-threaded.
- Scale out, not up. Past ~1.3 CPUs a pod has nothing to spend the extra on: the loop thread alone is ~5 % of a core per call, and turn latency held flat to 8 calls on 2 CPUs while underruns did not.
- Memory: ~265 MiB once the first call has loaded Smart Turn, + ~45 MiB per call (355 MiB at 2 calls, 627 MiB at 8) — 1 GiB covers the target with plenty of headroom. The Smart Turn model is one ONNX session per process, shared by every call; a call keeps its own Silero model (~5 MiB), pipeline and recording buffer. Before that (#271) each call loaded its own session, ~123 MiB a call: 36 MiB of weights plus the inference buffers ONNX Runtime keeps per session (~20 MiB more after its first inference), so the saving per call is about twice the model file.
turncall_active_calls behind autoscaling.enabled.
What was measured
make load-test (scripts/calls_per_pod/run.sh) runs the real app in the
API image under a pod’s CPU/memory budget, with the test harness’s
stub STT/LLM/TTS and fake carrier, and a load generator in a second container.
Each step opens N Twilio Media Streams; every caller plays recorded speech
(macOS say, 8 kHz — the harness’s test tone is not speech to Silero, and VAD
plus Smart Turn are most of a turn’s work) in real time, waits for the agent’s
answer (a 12-word reply at 400 ms a word), pauses 0.5–1.5 s, and speaks again.
Stubs answer instantly, so the measurement is TurnCall’s own per-call work —
VAD, Smart Turn, resampling, serialization, recording, database writes — and
not a provider’s rate limit.
Per step (2 minutes measured, after a 10 s ramp and 8 s warm-up) it reports:
A step is ok while every call connects and is answered, underruns stay at or
below 1 per minute of agent audio, and p95 turn latency stays within 300 ms of
the first step’s median.
capacity is the largest N with every step up to it ok.
Machine and conditions
- Host: Apple M1 Pro (8 cores, 6 performance + 2 efficiency), 16 GB RAM, macOS 26.5.2.
- Docker Desktop VM: Linux 6.12 aarch64, 8 vCPU, 7.75 GiB. Image
python:3.12-slim, pipecat 1.12.0, onnxruntime 1.24.4, torch 2.14.1+cpu, uvloop. Pod--memory 2g; load generator--cpus 3, one process per 20 calls. - Noisy. Other agents were running test suites on the host throughout: load average 5–19 on 8 cores, host CPU idle often 3–50 %, plus two dev containers using ~0.4 CPU in the VM. An idle pod’s loop lag was p99 10 ms / max 39 ms here (about 1 ms on a quiet machine), which is why even 1–2 call steps show 10–50 ms p99 lag. The figures are a floor for an x86 node with nothing else on its core.
- aarch64, not x86. ONNX inference speed differs per architecture; rerun on your node type before relying on the number.
- Before the live-call limit. These figures predate the live-call lease
(#224):
make load-testnow gives each call its lease in the stack’s Redis, as production does — one Redis round trip at the start and every 30 s after. The load project sets no limit, so no step is refused for being over one. - Not included: OpenTelemetry span export (tracing was off; it costs CPU per turn), ambience mixing, knowledge-base retrieval, MCP, post-call analysis (runs after the call, on a provider).
Raw results
--cpus 1 (quota):
--cpuset-cpus, no quota):
--cpus 2 (quota):
rssMB is the old ~115 MiB a call.
Full garbage collections
From 6 calls up, a full collection ran about once in two minutes and paused the process for 300–800 ms — every call’s audio at once, which was the longest single gap in each of those steps. It scans every object alive in the process, and most of those (pipecat and its imports) were allocated at startup and never die. So the app moves everything alive at the end of startup out of the collector’s reach (gc.freeze(), #270). Three runs back to back on one
dedicated core — the code before #271, with the shared Smart Turn session, and
with both — gave:
* client-bound, unreliable. That frozen run had no full collection at 6 or 8
calls, and the longest gap a caller heard there fell to ~70 ms. Full
collections are shorter, not gone: a re-run of the same code had them at 4 and
10 calls (61 and 167 ms), and once at 749 ms on the 8-call step, where the host
was overloaded — the load generator fell behind and the VM used 2.6 CPUs, which
stretches a pause’s wall-clock time:
Run it yourself
scripts/calls_per_pod/results/ (a JSON per run plus the pod’s
log). IMAGE, NETWORK and PG_CONTAINER point it at another stack; the
script makes and migrates its own database (LOAD_DB, default turncall_load),
since calls write real rows.