> ## Documentation Index
> Fetch the complete documentation index at: https://docs.turncall.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Create Eval Scenario

> Create a scenario. The definition is validated by round-tripping it
through pipecat's parser and then stored verbatim.



## OpenAPI

````yaml /openapi.json post /v1/eval-scenarios
openapi: 3.1.0
info:
  title: TurnCall
  description: Production voice agent platform API
  version: 0.1.0
servers: []
security: []
paths:
  /v1/eval-scenarios:
    post:
      tags:
        - evals
      summary: Create Eval Scenario
      description: |-
        Create a scenario. The definition is validated by round-tripping it
        through pipecat's parser and then stored verbatim.
      operationId: create_eval_scenario_v1_eval_scenarios_post
      parameters:
        - name: authorization
          in: header
          required: false
          schema:
            anyOf:
              - type: string
              - type: 'null'
            title: Authorization
      requestBody:
        required: true
        content:
          application/json:
            schema:
              $ref: '#/components/schemas/CreateEvalScenarioRequest'
      responses:
        '201':
          description: Successful Response
          content:
            application/json:
              schema:
                type: object
                additionalProperties: true
                title: Response Create Eval Scenario V1 Eval Scenarios Post
        '422':
          description: Validation Error
          content:
            application/json:
              schema:
                $ref: '#/components/schemas/HTTPValidationError'
components:
  schemas:
    CreateEvalScenarioRequest:
      properties:
        name:
          type: string
          maxLength: 255
          minLength: 1
          title: Name
        description:
          anyOf:
            - type: string
              maxLength: 2000
            - type: 'null'
          title: Description
        definition:
          additionalProperties: true
          type: object
          title: Definition
        tool_mocks:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Tool Mocks
        tool_policy:
          anyOf:
            - $ref: '#/components/schemas/EvalToolPolicy'
            - type: 'null'
        tags:
          items:
            type: string
          type: array
          maxItems: 32
          title: Tags
        default_target:
          anyOf:
            - additionalProperties: true
              type: object
            - type: 'null'
          title: Default Target
        judge:
          anyOf:
            - $ref: '#/components/schemas/EvalModelSchema'
            - type: 'null'
        simulator:
          anyOf:
            - $ref: '#/components/schemas/EvalModelSchema'
            - type: 'null'
      type: object
      required:
        - name
        - definition
      title: CreateEvalScenarioRequest
    HTTPValidationError:
      properties:
        detail:
          items:
            $ref: '#/components/schemas/ValidationError'
          type: array
          title: Detail
      type: object
      title: HTTPValidationError
    EvalToolPolicy:
      type: string
      enum:
        - mock_only
        - live
      title: EvalToolPolicy
      description: >-
        What happens when the agent calls a tool with no mock (slice #71).


        `mock_only` is the default and fails closed: a scenario pointed at an
        agent

        whose tools have real side effects must not book a real appointment on

        every iteration. `live` is the explicit opt-in for read-only lookups.
    EvalModelSchema:
      properties:
        provider:
          $ref: '#/components/schemas/EvalModelProvider'
          default: ollama
        model:
          anyOf:
            - type: string
              maxLength: 255
            - type: 'null'
          title: Model
        temperature:
          anyOf:
            - type: number
              maximum: 2
              minimum: 0
            - type: 'null'
          title: Temperature
        endpoint:
          anyOf:
            - type: string
              maxLength: 2048
            - type: 'null'
          title: Endpoint
      additionalProperties: false
      type: object
      title: EvalModelSchema
      description: >-
        One LLM an eval runs besides the agent's: the judge, or the persona
        (#118).


        `provider` is a closed set mapping to factories TurnCall ships

        (`evals.judges.PROVIDERS`). Pipecat's own escape hatch is `factory`, a

        dotted path it hands to `importlib.import_module` — safe for a file on

        disk, remote code execution for a request body — so a caller names a

        provider and never a path. `extra="forbid"` is what makes that a rule

        rather than a convention: a request carrying `factory` is a 422.
    ValidationError:
      properties:
        loc:
          items:
            anyOf:
              - type: string
              - type: integer
          type: array
          title: Location
        msg:
          type: string
          title: Message
        type:
          type: string
          title: Error Type
        input:
          title: Input
        ctx:
          type: object
          title: Context
      type: object
      required:
        - loc
        - msg
        - type
      title: ValidationError
    EvalModelProvider:
      type: string
      enum:
        - ollama
        - openai
        - anthropic
      title: EvalModelProvider
      description: >-
        Who runs an eval's judge, or its persona (#118).


        A closed set on purpose. Pipecat reaches anything but Ollama through

        `factory`, a dotted path it imports — so the provider name is what a

        request carries, and `evals.judges.PROVIDERS` maps it to a callable

        TurnCall ships. The set is the allowlist.


        `ollama` stays the default: it is pipecat's, it is local, and a judge
        that

        needs no key is what keeps an eval runnable on a laptop.

````