Skip to main content
POST
Create Eval Scenario

Headers

authorization
string | null

Body

application/json
name
string
required
Required string length: 1 - 255
definition
Definition · object
required
description
string | null
Maximum string length: 2000
tool_mocks
Tool Mocks · object | null
tool_policy
enum<string> | null

What happens when the agent calls a tool with no mock (slice #71).

mock_only is the default and fails closed: a scenario pointed at an agent whose tools have real side effects must not book a real appointment on every iteration. live is the explicit opt-in for read-only lookups.

Available options:
mock_only,
live
tags
string[]
Maximum array length: 32
default_target
Default Target · object | null
judge
EvalModelSchema · object | null

One LLM an eval runs besides the agent's: the judge, or the persona (#118).

provider is a closed set mapping to factories TurnCall ships (evals.judges.PROVIDERS). Pipecat's own escape hatch is factory, a dotted path it hands to importlib.import_module — safe for a file on disk, remote code execution for a request body — so a caller names a provider and never a path. extra="forbid" is what makes that a rule rather than a convention: a request carrying factory is a 422.

simulator
EvalModelSchema · object | null

One LLM an eval runs besides the agent's: the judge, or the persona (#118).

provider is a closed set mapping to factories TurnCall ships (evals.judges.PROVIDERS). Pipecat's own escape hatch is factory, a dotted path it hands to importlib.import_module — safe for a file on disk, remote code execution for a request body — so a caller names a provider and never a path. extra="forbid" is what makes that a rule rather than a convention: a request carrying factory is a 422.

Response

Successful Response

The response is of type Response Create Eval Scenario V1 Eval Scenarios Post · object.