Create Eval Scenario
Create a scenario. The definition is validated by round-tripping it through pipecat’s parser and then stored verbatim.
Headers
Body
1 - 2552000What happens when the agent calls a tool with no mock (slice #71).
mock_only is the default and fails closed: a scenario pointed at an agent
whose tools have real side effects must not book a real appointment on
every iteration. live is the explicit opt-in for read-only lookups.
mock_only, live 32One LLM an eval runs besides the agent's: the judge, or the persona (#118).
provider is a closed set mapping to factories TurnCall ships
(evals.judges.PROVIDERS). Pipecat's own escape hatch is factory, a
dotted path it hands to importlib.import_module — safe for a file on
disk, remote code execution for a request body — so a caller names a
provider and never a path. extra="forbid" is what makes that a rule
rather than a convention: a request carrying factory is a 422.
One LLM an eval runs besides the agent's: the judge, or the persona (#118).
provider is a closed set mapping to factories TurnCall ships
(evals.judges.PROVIDERS). Pipecat's own escape hatch is factory, a
dotted path it hands to importlib.import_module — safe for a file on
disk, remote code execution for a request body — so a caller names a
provider and never a path. extra="forbid" is what makes that a rule
rather than a convention: a request carrying factory is a 422.
Response
Successful Response
The response is of type Response Create Eval Scenario V1 Eval Scenarios Post · object.