Skip to main content
Voice agents are probabilistic. A prompt edit that reads better can quietly stop the agent from asking for a booking reference, and nothing fails — the call just goes worse. Evals let you state what an agent must do and have TurnCall check it, on every change, before a caller finds out. An eval scenario is one saved test. A run executes it against a target over N iterations. There are two kinds:

Scripted

A fixed conversation with per-turn expectations. Answers “at this point, did the agent make the right next decision?”

Simulation

A persona, a goal and success criteria — an LLM plays the caller and improvises. Answers “by the end, did it reach the right outcome?”

​
Your first scenario

A scenario’s definition is a conversation and what you expect back.
Then run it:
The response is 202 Accepted with a batch_id: a dedicated worker executes runs, never the API process, so eval load can never become dead air on a live call.
Assert content, not just the event. {"event": "llm_response"} on its own can pass with the agent’s LLM completely broken — when a provider rejects the model name, an empty response is still emitted. Always pair the event with text_contains, text_excludes, an eval: criterion, or a function_call with its calls:.There is no matches: field. Pipecat’s parser drops keys it does not know, so an expectation resting on one parses cleanly, stores, runs, and asserts nothing — which is why a create, an update and a run all report scenario_cannot_fail when nothing in a scenario asserts content.

​
Targets: what you run against

An agent row is an immutable version, so pinning by id means your scenario stops testing production the moment someone publishes the next version. agent_name follows the publish. inline runs a configuration that was never published — which is how you check a prompt change before shipping it, and how you point a scenario at sandbox tools.

​
Tool mocking, and why it’s on by default

A scenario that exercises “book the appointment” against an agent with a real webhook books a real appointment, on every iteration. So scenarios carry mocks:
A mocked tool returns your canned response and the real webhook is never called — webhook tools, MCP tools, built-ins and query_knowledge (a tool-mode knowledge base’s search) alike. Under mock_only, the default, a tool call with no mock is refused and the run ends as errored naming the tool, rather than executing. live is the opt-in for agents whose tools are read-only lookups. load_skill is the exception: it is never mocked. Loading a skill adds the skill’s instructions and tools to the agent under test and reaches nothing outside it, so it always runs, also under mock_only. Its entry in tool_calls has outcome applied, and a mock for it is ignored and reported as a mock_ignored warning. The skill’s own tools are ordinary tools: mock them, or they are refused. A function_call expectation on load_skill asserts that the agent loaded a skill:
move_to_step is never mocked either. A move in a procedure changes where the agent under test is in its procedure and reaches nothing outside it, so it always runs, the procedure accepting or refusing it as on a call. A canned “accepted” would hide exactly the skipped step the scenario is there to catch, so a mock for it is ignored with the same mock_ignored warning. A move is a tool call, so a function_call expectation asserts the path, one move per turn, and the iteration’s tool_calls show the path taken (every move applied, an accepted or refused one alike, the result saying which):
An agent that skips the confirmation and moves straight to cancelled fails this run: its move arrives with the wrong arguments (and the procedure refuses it). That is a verdict about the agent, so the run is failed, not errored. An eval records no procedure.* call events; the path is in tool_calls. A move onto an escalate step runs its transfer or handoff as transfer_call or handoff_to_agent would run: mocked by a mock of that name, and refused under mock_only without one, so a scenario that reaches an escalate step mocks its action. Mocks belong to the scenario, not the run: “the booking succeeds, does it report the confirmation correctly?” and “the booking fails, does it avoid claiming success?” are two different tests.
examples/eval-tool-mock runs all three cases against an agent whose booking webhook points at a host that does not exist — so a passing run is proof the webhook was never called, not a claim.

​
Text or audio

modality: "text" skips STT and TTS — fast, silent, cheap, and what you run on every pull request. modality: "audio" synthesizes the caller’s turns into the agent’s real STT and judges a transcription of the agent’s real TTS. Audio is what covers recognition, synthesis and turn timing; text cannot see any of it.
Audio runs use local models for the caller’s voice and the judge’s transcription, downloaded on first use. Budget for that the first time.

​
Choosing the judge

An eval: criterion and every simulation verdict are decided by a judge LLM. Left alone that is pipecat’s own — a local Ollama — which is why an eval: assertion errors on a machine with no Ollama running, while text_contains and function_call build no judge at all. A scenario can name its own:
provider is a closed set — ollama, openai, anthropic, jev — mapping to factories TurnCall ships. Pipecat’s own escape hatch is factory, a dotted path it hands to importlib.import_module: fine for a file on your disk, remote code execution for a request body. A definition naming one anywhere pipecat reads it (judge.eval, judge.transcription, simulator, user.speech) is refused at the API boundary, and api_key is not a field — the key comes from the same place the agent’s own LLM’s does: the project’s credential for that provider, else the platform environment (see API keys). model empty means that provider’s default. temperature has no pipecat field and rides in extra; it is unset by default, because a verdict that varies run to run is not a verdict. It is dropped for models that reject it — Anthropic, and OpenAI’s o1/o3/o4/gpt-5 families — by the rule the call path already follows.

​
Jev, a judge that is not an LLM

jev is TypeSafe’s classifier: a few hundred milliseconds per verdict, with a calibrated probability instead of a model’s opinion of its own certainty. It needs a TypeSafe key — the project’s typesafe credential, else TYPESAFE_API_KEY — and the evals-jev extra — neither checked when the scenario is created, because both are properties of a deployment and a stored scenario outlives the one it was written against. The run fails, naming which of the two is missing. It is judge-only, and structurally so: the allowlist is split by role, because a simulator is linked into a live pipeline to speak and a classifier has nothing to link. simulator: {"provider": "jev"} is refused with the role in the message. It takes no temperature either — a classifier is not a sampler.

​
Who writes the reason

Since pipecat 1.12 the verdict itself comes from a classifier, which gives no prose. A second model — the explainer — writes the reason behind every no and every verdict the judge was unsure of. For an LLM-backed judge, pipecat explains with the judging model, and TurnCall leaves that alone. For a classifier-only judge (jev) there is no such model, and pipecat’s fallback is its own default LLM service — a local Ollama, so a judge pointed at a hosted vendor would silently acquire a dependency on a service nobody provisioned and report it as an unreachable connection mid-run. TurnCall compiles the explainer from the platform’s configured judge instead, and where the platform has configured none it turns reasons off explicitly. A verdict with probabilities and no prose beats one that fails on a service nobody set up. A raw explainer: inside a stored definition still wins, like the rest of that block. harness_config records explainer_used, explainer_provider and explainer_model.

​
Set it once

Three environment variables on the worker, and the matching EVAL_SIMULATOR_* trio:
Precedence is narrowest-wins: the scenario’s block, then these, then pipecat’s Ollama. A block is taken whole, never merged field by field — a platform model landing on a scenario’s provider would name a model that provider has never heard of. Sending "judge": null on an update clears a scenario’s own block and returns it to the default. There is deliberately no run-level override: a judge on POST /v1/eval-runs is a 422. A run is what it was queued as, and two runs of one scenario decided by different judges are incomparable with nothing on either row explaining why — the same reason mocks belong to the scenario.

​
Knowing which judge decided

Every run’s harness_config records judge_provider, judge_model, judge_temperature, judge_service, judge_factory and judge_used — plus the explainer trio above — the last because pipecat fills its default in whether or not a judge is ever asked anything, and a snapshot naming a model that never ran is the same lie as a policy that was never enforced. On top of that, a run whose judge differs from the last run of the same scenario to reach a verdict carries a judge_changed warning naming both. A scenario that passed yesterday and fails today otherwise reads identically whether the agent regressed or the judge moved underneath it. The same snapshot records worker_version, worker_started_at and worker_stale — the last true when the worker’s source has been edited since the process started, which is the mounted-development case where a scenario’s judge is set, stored correctly, and quietly ignored by a process that predates the feature. A stale worker adds a worker_stale warning saying when each happened.

​
Simulations

Give a persona and a goal instead of turns, and an LLM improvises the call:
The judge rules on success once over the whole conversation; metrics are scored per agent turn, and one below its min_score fails the iteration. A persona never says the same thing twice, so use iterations — one run is an anecdote.

​
In CI

Exit 0 only if every run passed. A failure exits 1; an error — a judge outage, a pipeline that died — exits 2, so a broken checker never reads as a broken agent.
turncall eval show <run_id> prints a run’s transcript with its verdicts. Scenario files are the API request body unchanged — turncall eval run scenarios/*.json --agent-name support runs them without storing anything.

​
Start from a real conversation

Evals only find what someone thought to test. Turn a conversation that already happened into a draft — a voice call:
or a text conversation — SMS, chat or WhatsApp:
Both produce the same thing. The caller’s utterances become turns, the agent’s replies become expectations, and every tool the agent called becomes both an expectation and a mock seeded with what that tool actually returned — so the draft is safe to run straight away. save: false is the default and returns the draft without storing it; true stores it. A session names its agent directly, so from-session takes its default_target from the session. A call may have run an inline agent with no agent row at all, so from-call reads the call record instead. To convert a WebRTC call the browser just made, read the call id from the signalling answer — POST /v1/webrtc/connect returns it beside sdp and type. See WebRTC.
It is a draft. The conversation is faithful, but the expectations assert whatever the agent did that day, mistakes included. Trim each text_contains to the part that actually matters before trusting a green run.

​
Results

call.ended has no equivalent here — subscribe to eval.run.started and eval.run.completed instead. The completed event is comprehensive: status, counts, every iteration’s transcript and failures, and the configuration snapshots. The envelope carries eval_run_id beside call_id and session_id. Aggregate a whole batch without pulling transcripts:
A run reports passed_count/failed_count out of iterations — there is no score. errored is neither: it means the harness could not complete, so it counts toward no rate at all. Every verdict a judge produced carries a confidence beside it — a scripted expectation and its failures, a simulation’s goal verdict, and each per-turn verdict of a judged metric. null is not zero: zero is “the judge was certain of the opposite”, the strongest verdict there is, while null means no judge made a claim at all (text_contains, a function_call, a measured metric). There is deliberately no aggregate — averaging per-turn probabilities into one number invents a figure nothing measured.
confidence reads null on every run today. Pipecat 1.12 carries it on its own verdict object and then drops it building the results TurnCall reads, so the field is wired and waiting: it starts reporting on the next pipecat bump that threads it through, with no change here.
The run’s resolved_config is the agent configuration that ran. For an agent with skills it also holds resolved_skills, the skill versions the run resolved before its first iteration ([{id, skill, version, pinned}]). Every iteration runs those versions, even if a skill is published again while the run is going. A run also carries warnings: things its author needs to know that are not verdicts. A mock keyed to a tool this run can never call, a mock that is ignored (load_skill, move_to_step), a skill that did not resolve, an agent whose real tools are allowed to fire, a scenario nothing in which can fail, a judge that changed since the last verdict, a worker older than the code it runs. They ride on the run, the batch summary and turncall eval show, because the person reading a green verdict is rarely the one who wrote the scenario.

​
What evals cannot see

Worth knowing before you trust a green suite. An eval swaps only the transport, so everything inside it is invisible: Everything the pipeline builds is covered, which is where the regressions that shipped have actually lived: STT and TTS construction, provider model names, VAD and turn-detection wiring, the tool bridge, knowledge retrieval. The judge is an LLM and is not deterministic. Use binary criteria, use iterations, and treat a single flip as a signal to look rather than a verdict — and check judge_changed before blaming the agent.