> ## Documentation Index > Fetch the complete documentation index at: https://docs.turncall.io/llms.txt > Use this file to discover all available pages before exploring further. # Create Eval Run > Queue a run per scenario. 202: the worker executes them, never this process. A `tag` fans out to every scenario carrying it; the runs share one batch id so a single request has a single readable verdict. Each run is queued independently, so one that cannot be pushed does not cost the others theirs. ## OpenAPI ````yaml /openapi.json post /v1/eval-runs openapi: 3.1.0 info: title: TurnCall description: Production voice agent platform API version: 0.1.0 servers: [] security: [] paths: /v1/eval-runs: post: tags: - evals summary: Create Eval Run description: >- Queue a run per scenario. 202: the worker executes them, never this process. A `tag` fans out to every scenario carrying it; the runs share one batch id so a single request has a single readable verdict. Each run is queued independently, so one that cannot be pushed does not cost the others theirs. operationId: create_eval_run_v1_eval_runs_post parameters: - name: authorization in: header required: false schema: anyOf: - type: string - type: 'null' title: Authorization requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/CreateEvalRunRequest' responses: '202': description: Successful Response content: application/json: schema: type: object additionalProperties: true title: Response Create Eval Run V1 Eval Runs Post '422': description: Validation Error content: application/json: schema: $ref: '#/components/schemas/HTTPValidationError' components: schemas: CreateEvalRunRequest: properties: scenario_id: anyOf: - type: string format: uuid - type: 'null' title: Scenario Id tag: anyOf: - type: string maxLength: 64 minLength: 1 - type: 'null' title: Tag scenario: anyOf: - $ref: '#/components/schemas/InlineScenario' - type: 'null' target: $ref: '#/components/schemas/EvalTarget' modality: $ref: '#/components/schemas/EvalModality' default: text iterations: type: integer minimum: 1 title: Iterations default: 1 additionalProperties: false type: object required: - target title: CreateEvalRunRequest description: >- Run one stored scenario, every scenario carrying a tag (#75), or a scenario supplied inline (#77). Exactly one of the three. A tag fans out to one run per matching scenario, all sharing a batch id, so one request has one readable verdict — which is what the CLI's single exit code is built on. `extra="forbid"` is what makes "no run-level judge" a rule (#119). A run is what it was queued as: a request that could swap the judge would make two runs of one scenario incomparable with nothing on either row explaining why, and quietly ignoring the field is the same outcome with the author believing otherwise. Name it on the scenario — including the inline one above, which is where a run-shaped `judge:` was aiming. HTTPValidationError: properties: detail: items: $ref: '#/components/schemas/ValidationError' type: array title: Detail type: object title: HTTPValidationError InlineScenario: properties: name: type: string maxLength: 255 minLength: 1 title: Name definition: additionalProperties: true type: object title: Definition tool_mocks: anyOf: - additionalProperties: true type: object - type: 'null' title: Tool Mocks tool_policy: anyOf: - $ref: '#/components/schemas/EvalToolPolicy' - type: 'null' judge: anyOf: - $ref: '#/components/schemas/EvalModelSchema' - type: 'null' simulator: anyOf: - $ref: '#/components/schemas/EvalModelSchema' - type: 'null' type: object required: - name - definition title: InlineScenario description: >- A scenario supplied on the run instead of stored first (#77). This is what a local file holds, and why the CLI needs no format of its own: the file is the API request body, validated by the same parser a stored scenario is. Nothing is written to `eval_scenarios` — the run's `scenario_id` is null and its `resolved_scenario` snapshot is the record, which is the same shape ADR-0017 uses for an inline agent. EvalTarget: properties: type: type: string pattern: ^(agent|agent_name|inline)$ title: Type agent_id: anyOf: - type: string format: uuid - type: 'null' title: Agent Id name: anyOf: - type: string - type: 'null' title: Name environment: anyOf: - type: string - type: 'null' title: Environment agent: anyOf: - additionalProperties: true type: object - type: 'null' title: Agent type: object required: - type title: EvalTarget description: >- What a run points at (#74). Three forms, and the difference between the first two matters: an agent row is one immutable version, so `agent` pins a version forever — a scenario targeting it silently stops testing production the moment the next version is published. `agent_name` resolves to whatever is published at run time. `inline` has no row at all, which is how a prompt or model change is evaluated *before* publishing it, and the sandbox a scenario is pointed at when its tools have real side effects. EvalModality: type: string enum: - text - audio title: EvalModality description: |- Whether the conversation is spoken or typed. `text` skips STT and TTS entirely, so it is fast and free and covers the smallest share of a voice platform's risk; `audio` is the only mode that reaches the provider-connect path where most regressions have lived. ValidationError: properties: loc: items: anyOf: - type: string - type: integer type: array title: Location msg: type: string title: Message type: type: string title: Error Type input: title: Input ctx: type: object title: Context type: object required: - loc - msg - type title: ValidationError EvalToolPolicy: type: string enum: - mock_only - live title: EvalToolPolicy description: >- What happens when the agent calls a tool with no mock (slice #71). `mock_only` is the default and fails closed: a scenario pointed at an agent whose tools have real side effects must not book a real appointment on every iteration. `live` is the explicit opt-in for read-only lookups. EvalModelSchema: properties: provider: $ref: '#/components/schemas/EvalModelProvider' default: ollama model: anyOf: - type: string maxLength: 255 - type: 'null' title: Model temperature: anyOf: - type: number maximum: 2 minimum: 0 - type: 'null' title: Temperature endpoint: anyOf: - type: string maxLength: 2048 - type: 'null' title: Endpoint additionalProperties: false type: object title: EvalModelSchema description: >- One LLM an eval runs besides the agent's: the judge, or the persona (#118). `provider` is a closed set mapping to factories TurnCall ships (`evals.judges.PROVIDERS`). Pipecat's own escape hatch is `factory`, a dotted path it hands to `importlib.import_module` — safe for a file on disk, remote code execution for a request body — so a caller names a provider and never a path. `extra="forbid"` is what makes that a rule rather than a convention: a request carrying `factory` is a 422. EvalModelProvider: type: string enum: - ollama - openai - anthropic title: EvalModelProvider description: >- Who runs an eval's judge, or its persona (#118). A closed set on purpose. Pipecat reaches anything but Ollama through `factory`, a dotted path it imports — so the provider name is what a request carries, and `evals.judges.PROVIDERS` maps it to a callable TurnCall ships. The set is the allowlist. `ollama` stays the default: it is pipecat's, it is local, and a judge that needs no key is what keeps an eval runnable on a laptop. ````