> ## Documentation Index > Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Create evaluator validation plan > Requires scopes: eval:read ## OpenAPI ````yaml /api-reference/openapi.json post /api/evaluator-validation/plan openapi: 3.1.2 info: title: EvalGate API version: 18.13.0 description: EvalGate API. See docs/api-contract.md for stability commitments. servers: - url: https://evalgate.com description: Production - url: http://localhost:3000 description: Local security: [] paths: /api/evaluator-validation/plan: post: tags: - evaluator validation summary: Create evaluator validation plan description: 'Requires scopes: eval:read' operationId: post_evaluator_validation_plan requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/PostApiEvaluatorValidationPlanRequest' responses: '200': description: Successful response content: application/json: schema: $ref: >- #/components/schemas/PostApiEvaluatorValidationPlanResponse200ApplicationJson '400': $ref: '#/components/responses/ValidationError' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/InternalError' '503': $ref: '#/components/responses/ServiceUnavailable' security: - bearerAuth: [] components: schemas: PostApiEvaluatorValidationPlanRequest: type: object properties: evaluatorReleaseId: type: string evaluationRunId: type: number datasetVersionId: type: string validationType: oneOf: - type: string enum: - calibration - type: string enum: - coverage - type: string enum: - full - type: string enum: - human_alignment - type: string enum: - pairwise_bias - type: string enum: - post_run - type: string enum: - predictive_validity - type: string enum: - repeatability requestedMetrics: type: array items: type: string samplingManifest: type: object properties: strategy: type: string seed: type: number populationReference: type: string strata: type: array items: type: string exclusions: type: array items: type: object properties: reason: type: string count: type: number additionalProperties: false required: - reason - count metadata: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' additionalProperties: false required: - strategy - seed idempotencyKey: type: string comparison: type: object properties: baselineValidationRunId: type: string purpose: oneOf: - type: string enum: - evaluator_comparison - type: string enum: - system_comparison taskId: type: string minimumReleaseSupport: type: number recallFloor: type: number unsafeAutoPassCeiling: type: number criticalSlices: type: array items: type: object properties: sliceId: type: string minimumSupport: type: number recallFloor: type: number unsafeAutoPassCeiling: type: number badCaseAutoPassCeiling: type: number additionalProperties: false required: - sliceId - minimumSupport - recallFloor - unsafeAutoPassCeiling - badCaseAutoPassCeiling requiredApprovalStage: oneOf: - type: string enum: - measured_result - type: string enum: - reference_protocol additionalProperties: false required: - baselineValidationRunId - purpose - taskId - minimumReleaseSupport - recallFloor - unsafeAutoPassCeiling qualification: type: object properties: partitionId: type: string reviewItemId: type: number additionalProperties: false required: - partitionId - reviewItemId evidenceAnalysis: type: object properties: kind: type: string enum: - auto_adjudication decisionIds: type: array items: type: number additionalProperties: false required: - kind - decisionIds workflow: oneOf: - type: object properties: kind: type: string enum: - retrieval method: type: string enum: - lexical topK: type: number cutoff: type: string documentIds: type: array items: type: number referenceVersion: type: string additionalProperties: false required: - kind - method - topK - cutoff - documentIds - referenceVersion - type: object properties: kind: type: string enum: - stateful_agent attemptIds: type: array items: type: string additionalProperties: false required: - kind - attemptIds - type: object properties: kind: type: string enum: - memory sessionId: type: string roundId: type: string additionalProperties: false required: - kind - sessionId - roundId - type: object properties: kind: type: string enum: - review_preparation itemIds: type: array items: type: number additionalProperties: false required: - kind - itemIds executionBudget: type: object properties: maxCostUsd: type: number maxCalls: type: number maxRuntimeMs: type: number maxInputBytes: type: number maxDailyCostUsd: type: number additionalProperties: false required: - maxCostUsd - maxCalls - maxRuntimeMs - maxInputBytes source: type: object properties: repositoryId: type: number rootPath: type: string targetSha: type: string scanRunId: type: string additionalProperties: false required: - repositoryId - rootPath - targetSha - scanRunId planHash: type: string additionalProperties: false required: - evaluatorReleaseId - validationType - requestedMetrics - samplingManifest PostApiEvaluatorValidationPlanResponse200ApplicationJson: type: object properties: plan: type: object properties: version: type: string organizationId: $ref: '#/components/schemas/OrganizationId' evaluatorReleaseId: type: string evaluatorIdentityHash: type: string evaluationRunId: type: number snapshotHash: type: string source: oneOf: - type: 'null' - type: object properties: repositoryId: type: number rootPath: type: string targetSha: type: string scanRunId: type: string additionalProperties: false required: - repositoryId - rootPath - targetSha - scanRunId sourceSnapshot: oneOf: - type: 'null' - type: object properties: scanRunId: type: string snapshotHash: type: string completeness: type: string warnings: type: array items: type: string intelligenceRunId: type: string impactPlanId: type: string impactPlanHash: type: string requiredEvaluations: type: array items: type: object properties: evaluationId: type: number reasonCodes: type: array items: type: string hardRequired: type: boolean additionalProperties: false required: - evaluationId - reasonCodes - hardRequired additionalProperties: false required: - scanRunId - snapshotHash - completeness - warnings - intelligenceRunId - impactPlanId - impactPlanHash - requiredEvaluations budget: type: object properties: maxCostUsd: type: number maxCalls: type: number maxRuntimeMs: type: number maxInputBytes: type: number maxDailyCostUsd: type: number additionalProperties: false required: - maxCostUsd - maxCalls - maxRuntimeMs - maxInputBytes deterministicOnly: type: boolean providerBudget: oneOf: - type: 'null' - type: object properties: childCallUpperBound: type: number physicalCallUpperBound: type: number estimateCostUsd: type: number providerPricingIds: type: array items: type: number prices: type: array items: type: object properties: id: type: number provider: type: string model: type: string inputPerMillion: type: number outputPerMillion: type: number additionalProperties: false required: - id - provider - model - inputPerMillion - outputPerMillion deterministicOnly: type: boolean requiresTransportEnforcement: type: boolean reviewedBounds: type: array items: type: object properties: providerPricingId: type: number bound: type: object properties: version: type: number enum: - 1 provider: type: string model: type: string maxInputBytes: type: number maxOutputTokens: type: number maxRequestCostUsd: type: number reviewedBy: type: string reviewedAt: type: string sourceReference: type: string expiresAt: type: string transportContracts: type: array items: type: object properties: version: type: number enum: - 1 organizationId: $ref: '#/components/schemas/OrganizationId' gatewayConfigId: type: string backend: type: string requestedProvider: type: string requestedModel: type: string endpoint: type: string proxyEndpoint: type: string model: type: string upstreams: type: array items: type: object properties: provider: type: string model: type: string additionalProperties: false required: - provider - model maxInputBytes: type: number maxOutputTokens: type: number maxResponseBytes: type: number maxProviderAttempts: type: number maxRequestCostUsd: type: number timeoutMs: type: number reviewedBy: type: string reviewedAt: type: string expiresAt: type: string sourceReference: type: string additionalProperties: false required: - version - organizationId - gatewayConfigId - backend - requestedProvider - requestedModel - endpoint - model - upstreams - maxInputBytes - maxOutputTokens - maxResponseBytes - maxProviderAttempts - maxRequestCostUsd - timeoutMs - reviewedBy - reviewedAt - expiresAt - sourceReference additionalProperties: false required: - version - provider - model - maxInputBytes - maxOutputTokens - maxRequestCostUsd - reviewedBy - reviewedAt - sourceReference - expiresAt additionalProperties: false required: - providerPricingId - bound additionalProperties: false required: - childCallUpperBound - estimateCostUsd - providerPricingIds - prices - deterministicOnly - requiresTransportEnforcement - reviewedBounds suites: oneOf: - type: array items: type: object properties: evaluationId: type: number reasonCodes: type: array items: type: string hardRequired: type: boolean required: type: boolean state: type: string caseIds: oneOf: - type: 'null' - type: array items: type: number additionalProperties: false required: - evaluationId - reasonCodes - hardRequired - required - state - caseIds - type: array items: type: object properties: evaluationId: type: number required: type: boolean reason: type: string caseIds: type: array items: type: number additionalProperties: false required: - evaluationId - required - reason - caseIds coverage: type: object properties: planned: type: number observed: type: number missingCaseIds: type: array items: type: number impactSelection: type: string repositoryWideCoverage: type: string additionalProperties: false required: - planned - observed - missingCaseIds - impactSelection - repositoryWideCoverage requestedMetrics: type: array items: type: string samplingManifest: type: object properties: strategy: type: string seed: type: number populationReference: type: string strata: type: array items: type: string exclusions: type: array items: type: object properties: reason: type: string count: type: number additionalProperties: false required: - reason - count metadata: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' additionalProperties: false required: - strategy - seed comparison: oneOf: - type: 'null' - type: object properties: baselineValidationRunId: type: string purpose: oneOf: - type: string enum: - evaluator_comparison - type: string enum: - system_comparison taskId: type: string minimumReleaseSupport: type: number recallFloor: type: number unsafeAutoPassCeiling: type: number criticalSlices: type: array items: type: object properties: sliceId: type: string minimumSupport: type: number recallFloor: type: number unsafeAutoPassCeiling: type: number badCaseAutoPassCeiling: type: number additionalProperties: false required: - sliceId - minimumSupport - recallFloor - unsafeAutoPassCeiling - badCaseAutoPassCeiling requiredApprovalStage: oneOf: - type: string enum: - measured_result - type: string enum: - reference_protocol additionalProperties: false required: - baselineValidationRunId - purpose - taskId - minimumReleaseSupport - recallFloor - unsafeAutoPassCeiling qualification: oneOf: - type: 'null' - type: object properties: partitionId: type: string reviewItemId: type: number additionalProperties: false required: - partitionId - reviewItemId workflow: oneOf: - type: 'null' - type: object properties: kind: type: string enum: - retrieval method: type: string enum: - lexical topK: type: number cutoff: type: string documentIds: type: array items: type: number referenceVersion: type: string additionalProperties: false required: - kind - method - topK - cutoff - documentIds - referenceVersion - type: object properties: kind: type: string enum: - stateful_agent attemptIds: type: array items: type: string additionalProperties: false required: - kind - attemptIds - type: object properties: kind: type: string enum: - memory sessionId: type: string roundId: type: string additionalProperties: false required: - kind - sessionId - roundId - type: object properties: kind: type: string enum: - review_preparation itemIds: type: array items: type: number additionalProperties: false required: - kind - itemIds evidenceAnalysis: oneOf: - type: 'null' - type: object properties: kind: type: string enum: - auto_adjudication decisionIds: type: array items: type: number additionalProperties: false required: - kind - decisionIds authority: type: string reasons: type: array items: type: string planHash: type: string additionalProperties: false required: - version - organizationId - evaluatorReleaseId - evaluatorIdentityHash - evaluationRunId - snapshotHash - source - sourceSnapshot - budget - deterministicOnly - providerBudget - suites - coverage - requestedMetrics - samplingManifest - comparison - qualification - workflow - evidenceAnalysis - authority - reasons - planHash additionalProperties: false required: - plan JsonValue: oneOf: - type: 'null' - type: boolean - type: number - type: string - type: array items: $ref: '#/components/schemas/JsonValue' - type: object additionalProperties: $ref: '#/components/schemas/JsonValue' OrganizationId: type: string format: uuid ApiError: type: object properties: error: type: object properties: code: type: string enum: - UNAUTHORIZED - FORBIDDEN - NOT_FOUND - INVALID_REFERENCE - VALIDATION_ERROR - RATE_LIMITED - CONFLICT - INTERNAL_ERROR - SERVICE_UNAVAILABLE - QUOTA_EXCEEDED - FEATURE_LIMIT_REACHED - NO_ORG_MEMBERSHIP - SHARE_REVOKED - SHARE_EXPIRED - SHARE_UNAVAILABLE - PROVIDER_KEY_MISSING - PROVIDER_KEY_EXPIRED - PROVIDER_RATE_LIMITED - PROVIDER_UNAUTHORIZED - PROVIDER_UNAVAILABLE - AGENT_HANDOFF_INVALID - AGENT_HANDOFF_EXPIRED - AGENT_HANDOFF_DENIED - AGENT_HANDOFF_CONSUMED - PAYLOAD_TOO_LARGE - METHOD_NOT_ALLOWED message: type: string details: {} requestId: type: string format: uuid redaction: type: object properties: applied: type: boolean enum: - true fieldCount: type: integer minimum: 0 secretCount: type: integer minimum: 0 required: - applied - fieldCount - secretCount required: - code - message required: - error responses: ValidationError: description: Validation error content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Unauthorized: description: Unauthorized content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string WWW-Authenticate: description: >- Bearer challenge. Agent-facing routes include the RFC 9728 resource_metadata URL. schema: type: string Forbidden: description: Forbidden content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string NotFound: description: Resource not found content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Conflict: description: Conflict content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string RateLimited: description: Rate limit exceeded content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Retry-After: description: Delay in seconds before retrying when present. schema: type: integer minimum: 1 InternalError: description: Internal server error content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string ServiceUnavailable: description: Service unavailable content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: EvalGate API key description: >- Organization-scoped EvalGate API key. Discover the supported agent scopes at /.well-known/oauth-protected-resource. Per-operation grants are listed in x-evalgate-required-scopes; OpenAPI bearer security requirement arrays remain empty as required for non-OAuth schemes. x-evalgate-scopes-supported: - eval:read - eval:write - objectives:read - objectives:write - runs:read - runs:write - traces:read - traces:write - exports:download - reports:write - prompt:publish - scorer:publish - red_team:read - red_team:run - red_team:approve - docs:read - agent:read - agent:execute - resources:read - resources:write - contexts:read - contexts:write - observations:write - checks:read - checks:run - standards:read - standards:write - standards:approve - grants:read - grants:write - admin:keys - admin:workspace - admin:org x-evalgate-protected-resource-metadata: https://www.evalgate.com/.well-known/oauth-protected-resource ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.