> ## Documentation Index > Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Create llm judge test > Requires scopes: eval:write ## OpenAPI ````yaml /api-reference/openapi.json post /api/llm-judge/test openapi: 3.1.2 info: title: EvalGate API version: 18.13.0 description: EvalGate API. See docs/api-contract.md for stability commitments. servers: - url: https://evalgate.com description: Production - url: http://localhost:3000 description: Local security: [] paths: /api/llm-judge/test: post: tags: - llm judge summary: Create llm judge test description: 'Requires scopes: eval:write' operationId: post_llm_judge_test requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/PostApiLlmJudgeTestRequest' responses: '200': description: Successful response content: application/json: schema: $ref: >- #/components/schemas/PostApiLlmJudgeTestResponse200ApplicationJson '400': $ref: '#/components/responses/ValidationError' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/InternalError' '503': $ref: '#/components/responses/ServiceUnavailable' security: - bearerAuth: [] components: schemas: PostApiLlmJudgeTestRequest: type: object properties: input: type: string output: type: string promptTemplate: type: string provider: oneOf: - type: string enum: - local - type: string enum: - cursor - type: string enum: - typesafe - type: string enum: - openai - type: string enum: - anthropic - type: string enum: - google - type: string enum: - xai - type: string enum: - windsurf - type: string enum: - moonshot model: type: string expectedOutput: type: string context: oneOf: - type: string - type: object additionalProperties: $ref: '#/components/schemas/JsonValue' aggregation: oneOf: - type: string enum: - all_pass - type: string enum: - any_pass - type: string enum: - weighted - type: string enum: - primary_fallback - type: string enum: - escalate_on_disagreement - type: string enum: - confidence_escalate criteria: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' criteriaList: type: array items: type: object properties: id: type: string title: type: string matchCriteria: type: string additionalProperties: false required: - id - title - matchCriteria settings: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' behavior: type: string taskType: type: string passThreshold: type: number judges: type: array items: type: object properties: type: oneOf: - type: string enum: - decision - type: string enum: - rule - type: string enum: - llm id: type: string purpose: type: string enum: - customer_evaluation provider: oneOf: - type: string enum: - local - type: string enum: - cursor - type: string enum: - typesafe - type: string enum: - openai - type: string enum: - anthropic - type: string enum: - google - type: string enum: - xai - type: string enum: - windsurf - type: string enum: - moonshot model: type: string temperature: type: number maxTokens: type: number seed: type: number threshold: type: number weight: type: number topP: type: number timeoutMs: type: number rule: oneOf: - type: string enum: - json_schema - type: string enum: - exact_match - type: string enum: - contains_expected passThreshold: type: number top_p: type: number calibrationVersion: type: string decisionPolicyVersion: type: string cacheEnabled: oneOf: - type: boolean enum: - false - type: boolean enum: - true questions: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' primaryQuestionId: type: string evidenceQuestionIds: type: array items: type: string aggregationPolicy: oneOf: - type: string enum: - all_pass - type: string enum: - any_pass - type: string enum: - primary_question pinnedQuestionSchemaHash: type: string releaseBlocking: oneOf: - type: boolean enum: - false - type: boolean enum: - true confidencePolicy: type: object properties: autoAdjudicateMin: type: number frontierEscalateMin: type: number critiqueRequired: oneOf: - type: boolean enum: - false - type: boolean enum: - true additionalProperties: false required: - autoAdjudicateMin - frontierEscalateMin additionalProperties: false instabilityThreshold: type: number stabilityRuns: type: number additionalProperties: false required: - input - output - promptTemplate PostApiLlmJudgeTestResponse200ApplicationJson: type: object properties: promptPreview: type: string judgement: type: object properties: judgeResultId: type: number judgeConfigHash: type: string comparabilityStatus: oneOf: - type: string enum: - not_comparable - type: string enum: - exact_comparable - type: string enum: - calibrated_comparable - type: string enum: - trend_discontinuity provider: oneOf: - type: string enum: - local - type: string enum: - cursor - type: string enum: - rule - type: string enum: - typesafe - type: string enum: - openai - type: string enum: - anthropic - type: string enum: - google - type: string enum: - xai - type: string enum: - windsurf - type: string enum: - moonshot - type: string enum: - aggregate model: type: string score: type: number passed: type: boolean latencyMs: type: number aggregation: oneOf: - type: string enum: - all_pass - type: string enum: - any_pass - type: string enum: - weighted - type: string enum: - primary_fallback - type: string enum: - escalate_on_disagreement - type: string enum: - confidence_escalate reasoning: type: string flags: type: array items: type: string signals: type: object additionalProperties: oneOf: - type: string - type: number - type: boolean enum: - false - type: boolean enum: - true judgeId: type: string retries: type: number parseStatus: oneOf: - type: string enum: - failed - type: string enum: - not_applicable - type: string enum: - ok - type: string enum: - retried judgeResults: type: array items: type: object properties: type: oneOf: - type: string enum: - decision - type: string enum: - rule - type: string enum: - llm - type: string enum: - hybrid latencyMs: type: number judgeId: type: string provider: oneOf: - type: string enum: - local - type: string enum: - cursor - type: string enum: - rule - type: string enum: - typesafe - type: string enum: - openai - type: string enum: - anthropic - type: string enum: - google - type: string enum: - xai - type: string enum: - windsurf - type: string enum: - moonshot - type: string enum: - aggregate model: type: string errorCode: type: string error: type: string result: type: object properties: provider: oneOf: - type: string enum: - local - type: string enum: - cursor - type: string enum: - rule - type: string enum: - typesafe - type: string enum: - openai - type: string enum: - anthropic - type: string enum: - google - type: string enum: - xai - type: string enum: - windsurf - type: string enum: - moonshot - type: string enum: - aggregate model: type: string score: type: number passed: type: boolean latencyMs: type: number reasoning: type: string signals: type: object additionalProperties: oneOf: - type: string - type: number - type: boolean enum: - false - type: boolean enum: - true judgeId: type: string retries: type: number parseStatus: oneOf: - type: string enum: - failed - type: string enum: - not_applicable - type: string enum: - ok - type: string enum: - retried confidence: type: number modelCallId: type: string decisions: type: array items: oneOf: - type: object properties: type: type: string enum: - choice confidence: type: number question: type: string questionId: type: string choice: type: string probabilities: type: object additionalProperties: type: number additionalProperties: false required: - type - confidence - question - questionId - choice - probabilities - type: object properties: type: type: string enum: - score confidence: type: number question: type: string score: type: number questionId: type: string probabilities: type: object additionalProperties: type: number legend: type: object additionalProperties: type: string additionalProperties: false required: - type - confidence - question - score - questionId - probabilities - type: object properties: type: type: string enum: - noul question: type: string questionId: type: string noul: type: number additionalProperties: false required: - type - question - questionId - noul tokenUsage: type: object properties: input: type: number output: type: number total: type: number additionalProperties: false required: - input - output modelProvenance: type: object properties: provider: oneOf: - type: string enum: - local - type: string enum: - cursor - type: string enum: - openai - type: string enum: - anthropic - type: string enum: - google - type: string enum: - xai - type: string enum: - windsurf - type: string enum: - moonshot model: type: string temperature: type: number prompt_hash: type: string top_p: type: number max_tokens: type: number seed: type: number additionalProperties: false required: - provider - model - temperature - prompt_hash - top_p - max_tokens cached: oneOf: - type: boolean enum: - false - type: boolean enum: - true judgeFingerprint: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' criterionResults: type: array items: type: object properties: title: type: string score: type: number passed: type: boolean reasoning: type: string criterionId: type: string additionalProperties: false required: - title - score - passed - reasoning - criterionId decisionProvenance: type: object properties: purpose: oneOf: - type: string enum: - platform_assistance - type: string enum: - customer_evaluation - type: string enum: - internal_validation provider: type: string enum: - typesafe model: type: string disposition: oneOf: - type: string enum: - pass - type: string enum: - abstain - type: string enum: - human_review - type: string enum: - fail - type: string enum: - insufficient_evidence - type: string enum: - escalate_frontier - type: string enum: - observe - type: string enum: - not_eligible - type: string enum: - environment_blocked fallbackUsed: type: boolean questionSchemaHash: type: string inputConstructionVersion: type: string modelVersion: type: string threshold: type: number inputDigest: type: string calibrationVersion: type: string decisionPolicyVersion: type: string fallbackReason: type: string additionalProperties: false required: - purpose - provider - model - disposition - fallbackUsed - questionSchemaHash - inputConstructionVersion additionalProperties: false required: - provider - model - score - passed - latencyMs - reasoning - signals - judgeId - retries - parseStatus modelCallId: type: string weight: type: number modelProvenance: type: object properties: provider: oneOf: - type: string enum: - local - type: string enum: - cursor - type: string enum: - openai - type: string enum: - anthropic - type: string enum: - google - type: string enum: - xai - type: string enum: - windsurf - type: string enum: - moonshot model: type: string temperature: type: number prompt_hash: type: string top_p: type: number max_tokens: type: number seed: type: number additionalProperties: false required: - provider - model - temperature - prompt_hash - top_p - max_tokens providerExecutionState: oneOf: - type: string enum: - cancelled - type: string enum: - policy_blocked - type: string enum: - executed - type: string enum: - timed_out - type: string enum: - cache_reused - type: string enum: - skipped_by_policy - type: string enum: - deferred_by_policy - type: string enum: - rate_limited - type: string enum: - provider_unavailable - type: string enum: - model_retired - type: string enum: - authentication_failed - type: string enum: - malformed_response additionalProperties: false required: - type - latencyMs - judgeId disagreement: type: object properties: rate: type: number reasoningDivergence: type: number outlierJudgeIds: type: array items: type: string requiresEscalation: type: boolean nonPassRate: type: number interJudgeRate: oneOf: - type: 'null' - type: number additionalProperties: false required: - rate - reasoningDivergence - outlierJudgeIds - requiresEscalation confidence: type: number totalCostUsd: type: number tokenUsage: type: object properties: input: type: number output: type: number total: type: number additionalProperties: false required: - input - output executionCompleteness: type: object properties: state: oneOf: - type: string enum: - unknown - type: string enum: - complete - type: string enum: - incomplete expected: type: number successful: type: number incomplete: type: number complete: oneOf: - type: boolean enum: - false - type: boolean enum: - true additionalProperties: false required: - state variance: type: number additionalProperties: false required: - provider - model - score - passed - latencyMs - aggregation - reasoning - flags - signals - judgeId - retries - parseStatus - judgeResults - disagreement additionalProperties: false required: - promptPreview - judgement JsonValue: oneOf: - type: 'null' - type: boolean - type: number - type: string - type: array items: $ref: '#/components/schemas/JsonValue' - type: object additionalProperties: $ref: '#/components/schemas/JsonValue' ApiError: type: object properties: error: type: object properties: code: type: string enum: - UNAUTHORIZED - FORBIDDEN - NOT_FOUND - INVALID_REFERENCE - VALIDATION_ERROR - RATE_LIMITED - CONFLICT - INTERNAL_ERROR - SERVICE_UNAVAILABLE - QUOTA_EXCEEDED - FEATURE_LIMIT_REACHED - NO_ORG_MEMBERSHIP - SHARE_REVOKED - SHARE_EXPIRED - SHARE_UNAVAILABLE - PROVIDER_KEY_MISSING - PROVIDER_KEY_EXPIRED - PROVIDER_RATE_LIMITED - PROVIDER_UNAUTHORIZED - PROVIDER_UNAVAILABLE - AGENT_HANDOFF_INVALID - AGENT_HANDOFF_EXPIRED - AGENT_HANDOFF_DENIED - AGENT_HANDOFF_CONSUMED - PAYLOAD_TOO_LARGE - METHOD_NOT_ALLOWED message: type: string details: {} requestId: type: string format: uuid redaction: type: object properties: applied: type: boolean enum: - true fieldCount: type: integer minimum: 0 secretCount: type: integer minimum: 0 required: - applied - fieldCount - secretCount required: - code - message required: - error responses: ValidationError: description: Validation error content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Unauthorized: description: Unauthorized content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string WWW-Authenticate: description: >- Bearer challenge. Agent-facing routes include the RFC 9728 resource_metadata URL. schema: type: string Forbidden: description: Forbidden content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string NotFound: description: Resource not found content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Conflict: description: Conflict content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string RateLimited: description: Rate limit exceeded content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Retry-After: description: Delay in seconds before retrying when present. schema: type: integer minimum: 1 InternalError: description: Internal server error content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string ServiceUnavailable: description: Service unavailable content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: EvalGate API key description: >- Organization-scoped EvalGate API key. Discover the supported agent scopes at /.well-known/oauth-protected-resource. Per-operation grants are listed in x-evalgate-required-scopes; OpenAPI bearer security requirement arrays remain empty as required for non-OAuth schemes. x-evalgate-scopes-supported: - eval:read - eval:write - objectives:read - objectives:write - runs:read - runs:write - traces:read - traces:write - exports:download - reports:write - prompt:publish - scorer:publish - red_team:read - red_team:run - red_team:approve - docs:read - agent:read - agent:execute - resources:read - resources:write - contexts:read - contexts:write - observations:write - checks:read - checks:run - standards:read - standards:write - standards:approve - grants:read - grants:write - admin:keys - admin:workspace - admin:org x-evalgate-protected-resource-metadata: https://www.evalgate.com/.well-known/oauth-protected-resource ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.