> ## Documentation Index > Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Create evalgate synthesize > Requires scopes: reports:write ## OpenAPI ````yaml /api-reference/openapi.json post /api/evalgate/synthesize openapi: 3.1.2 info: title: EvalGate API version: 18.13.0 description: EvalGate API. See docs/api-contract.md for stability commitments. servers: - url: https://evalgate.com description: Production - url: http://localhost:3000 description: Local security: [] paths: /api/evalgate/synthesize: post: tags: - evalgate summary: Create evalgate synthesize description: 'Requires scopes: reports:write' operationId: post_evalgate_synthesize parameters: - name: Idempotency-Key in: header required: false description: Makes retries safe by returning the original operation result. schema: type: string minLength: 1 maxLength: 255 requestBody: required: true content: application/json: schema: $ref: '#/components/schemas/PostApiEvalgateSynthesizeRequest' responses: '200': description: Successful response content: application/json: schema: $ref: >- #/components/schemas/PostApiEvalgateSynthesizeResponse200ApplicationJson '201': description: Successful response content: application/json: schema: $ref: >- #/components/schemas/PostApiEvalgateSynthesizeResponse201ApplicationJson '202': description: Successful response content: application/json: schema: $ref: >- #/components/schemas/PostApiEvalgateSynthesizeResponse202ApplicationJson '400': $ref: '#/components/responses/ValidationError' '401': $ref: '#/components/responses/Unauthorized' '403': $ref: '#/components/responses/Forbidden' '404': $ref: '#/components/responses/NotFound' '409': $ref: '#/components/responses/Conflict' '429': $ref: '#/components/responses/RateLimited' '500': $ref: '#/components/responses/InternalError' '503': $ref: '#/components/responses/ServiceUnavailable' security: - bearerAuth: [] components: schemas: PostApiEvalgateSynthesizeRequest: oneOf: - type: object properties: action: oneOf: - type: string enum: - review - type: string enum: - retry - type: string enum: - cancel - type: string enum: - review_bulk - type: string enum: - publish - type: string enum: - regenerate runId: type: string itemId: type: string decision: oneOf: - type: string enum: - accepted - type: string enum: - rejected rationale: type: string failureModes: type: array items: type: string itemIds: type: array items: type: string edits: type: object properties: input: type: string expectedOutput: oneOf: - type: 'null' - type: string failureMode: oneOf: - type: 'null' - type: string dimensions: type: object additionalProperties: type: string additionalProperties: false additionalProperties: false required: - action - runId - type: object properties: evaluationId: type: number source: type: object properties: type: oneOf: - type: string enum: - dataset - type: string enum: - evaluation_run - type: string enum: - trace - type: string enum: - dataset_content - type: string enum: - failure_cluster evaluationRunId: type: number datasetId: type: string traceIds: type: array items: type: number failureClusterIds: type: array items: type: string additionalProperties: false required: - type budget: type: object properties: maxTokens: type: number maxUsd: type: number maxItems: type: number additionalProperties: false idempotencyKey: type: string datasetId: type: string seed: type: string dimensions: type: object additionalProperties: type: array items: type: string count: oneOf: - type: 'null' - type: number rowSchema: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' executionMode: oneOf: - type: string enum: - preview - type: string enum: - batch execution: type: object properties: enabled: oneOf: - type: boolean enum: - false - type: boolean enum: - true required: oneOf: - type: boolean enum: - false - type: boolean enum: - true timeoutMs: type: number minimumScore: type: number additionalProperties: false modelPolicy: type: object properties: mode: oneOf: - type: string enum: - deterministic - type: string enum: - llm provider: type: string model: type: string temperature: type: number maxTokens: type: number systemPrompt: type: string additionalProperties: false failureModes: type: array items: type: string datasetContent: type: string distributionTargets: type: object additionalProperties: type: object additionalProperties: type: number maxCombinations: type: number stageMissionProposals: oneOf: - type: boolean enum: - false - type: boolean enum: - true publishDatasetVersion: oneOf: - type: boolean enum: - false - type: boolean enum: - true additionalProperties: false PostApiEvalgateSynthesizeResponse200ApplicationJson: oneOf: - oneOf: - type: 'null' - type: object properties: items: type: array items: type: object properties: id: type: string organizationId: $ref: '#/components/schemas/OrganizationId' generationRunId: type: string stableItemId: type: string ordinal: type: number status: type: string input: type: string expectedOutput: oneOf: - type: 'null' - type: string actual: oneOf: - type: 'null' - type: string failureMode: oneOf: - type: 'null' - type: string dimensions: type: object additionalProperties: type: string sourceCaseIds: type: array items: type: string dedupeHash: type: string provenance: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' validation: oneOf: - type: 'null' - type: object properties: status: oneOf: - type: string enum: - unknown - type: string enum: - pass - type: string enum: - fail issues: type: array items: type: object properties: path: type: string message: type: string additionalProperties: false required: - path - message checks: type: object additionalProperties: oneOf: - type: string enum: - unknown - type: string enum: - pass - type: string enum: - fail validatedAt: type: string additionalProperties: false required: - status - issues - checks - validatedAt execution: oneOf: - type: 'null' - type: object properties: status: oneOf: - type: string enum: - pending - type: string enum: - pass - type: string enum: - fail - type: string enum: - skipped output: oneOf: - type: 'null' - type: string score: oneOf: - type: 'null' - type: number error: oneOf: - type: 'null' - type: string latencyMs: oneOf: - type: 'null' - type: number model: oneOf: - type: 'null' - type: string provider: oneOf: - type: 'null' - type: string modelCallId: oneOf: - type: 'null' - type: string costUsd: oneOf: - type: 'null' - type: number tokens: oneOf: - type: 'null' - type: object properties: input: type: number output: type: number additionalProperties: false required: - input - output checkedAt: type: string additionalProperties: false required: - status - checkedAt attempt: type: number reviewDecision: oneOf: - type: 'null' - type: string reviewRationale: oneOf: - type: 'null' - type: string reviewerId: oneOf: - type: 'null' - type: string reviewedAt: oneOf: - type: 'null' - type: string format: date-time errorMessage: oneOf: - type: 'null' - type: string createdAt: type: string format: date-time updatedAt: type: string format: date-time additionalProperties: false required: - id - organizationId - generationRunId - stableItemId - ordinal - status - input - expectedOutput - actual - failureMode - dimensions - sourceCaseIds - dedupeHash - provenance - validation - execution - attempt - reviewDecision - reviewRationale - reviewerId - reviewedAt - errorMessage - createdAt - updatedAt nextItemCursor: oneOf: - type: 'null' - type: string recipe: type: object properties: source: type: object properties: type: oneOf: - type: string enum: - dataset - type: string enum: - evaluation_run - type: string enum: - trace - type: string enum: - dataset_content - type: string enum: - failure_cluster evaluationRunId: type: number datasetId: type: string traceIds: type: array items: type: number failureClusterIds: type: array items: type: string additionalProperties: false required: - type schema: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' distributionTargets: type: object additionalProperties: type: object additionalProperties: type: number dimensions: type: object additionalProperties: type: array items: type: string failureModes: type: array items: type: string count: type: number maxCombinations: type: number seed: type: string evaluationId: type: number datasetId: type: string stageMissionProposals: type: boolean publishDatasetVersion: type: boolean modelPolicy: type: object properties: mode: oneOf: - type: string enum: - deterministic - type: string enum: - llm provider: type: string model: type: string systemPrompt: type: string temperature: type: number maxTokens: type: number additionalProperties: false required: - mode budget: type: object properties: maxTokens: type: number maxUsd: type: number maxItems: type: number additionalProperties: false execution: type: object properties: enabled: type: boolean required: type: boolean minimumScore: type: number timeoutMs: type: number additionalProperties: false required: - enabled - required - minimumScore - timeoutMs additionalProperties: false required: - source - dimensions - failureModes - count - maxCombinations - seed - stageMissionProposals - publishDatasetVersion - modelPolicy - budget - execution id: type: string organizationId: $ref: '#/components/schemas/OrganizationId' evaluationId: oneOf: - type: 'null' - type: number datasetId: oneOf: - type: 'null' - type: string createdBy: oneOf: - type: 'null' - type: string idempotencyKey: type: string recipeHash: type: string sourceSnapshotHash: type: string sourceRefs: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' modelPolicy: type: object properties: mode: oneOf: - type: string enum: - deterministic - type: string enum: - llm provider: type: string model: type: string systemPrompt: type: string temperature: type: number maxTokens: type: number additionalProperties: false required: - mode budgetPolicy: type: object properties: maxTokens: type: number maxUsd: type: number maxItems: type: number additionalProperties: false lineage: type: object properties: sourceRefs: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' modelCalls: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' provider: oneOf: - type: 'null' - type: string model: oneOf: - type: 'null' - type: string configHash: oneOf: - type: 'null' - type: string budgetReservedUsd: type: number budgetConsumedUsd: type: number budgetConsumedTokens: type: number additionalProperties: false required: - sourceRefs - modelCalls qualitySummary: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' consumedCostUsd: type: string consumedTokens: type: number status: type: string totalCount: type: number pendingCount: type: number validCount: type: number invalidCount: type: number duplicateCount: type: number reviewedCount: type: number missionId: oneOf: - type: 'null' - type: string publishedDatasetVersionId: oneOf: - type: 'null' - type: string errorMessage: oneOf: - type: 'null' - type: string createdAt: type: string format: date-time updatedAt: type: string format: date-time completedAt: oneOf: - type: 'null' - type: string format: date-time additionalProperties: false required: - recipe - id - organizationId - evaluationId - datasetId - createdBy - idempotencyKey - recipeHash - sourceSnapshotHash - sourceRefs - modelPolicy - budgetPolicy - lineage - qualitySummary - consumedCostUsd - consumedTokens - status - totalCount - pendingCount - validCount - invalidCount - duplicateCount - reviewedCount - missionId - publishedDatasetVersionId - errorMessage - createdAt - updatedAt - completedAt - type: object properties: missionId: type: string proposalsInserted: type: number continuation: type: object properties: entryPoint: oneOf: - type: string enum: - repository - type: string enum: - playground - type: string enum: - mcp - type: string enum: - trace - type: string enum: - synthetic - type: string enum: - application-health - type: string enum: - auto - type: string enum: - harness workspace: oneOf: - type: string enum: - local - type: string enum: - evidence - type: string enum: - improve - type: string enum: - connect steps: type: array items: type: object properties: href: type: string id: type: string label: type: string description: type: string command: type: string additionalProperties: false required: - href - id - label - description additionalProperties: false required: - entryPoint - workspace - steps sourceCases: type: number sourceFailures: type: number selectedFailureModes: type: array items: type: string dimensionNames: type: array items: type: string dimensionCombinationCount: type: number dimensionCartesianProductTotal: type: number generated: type: number modeCounts: type: array items: type: object properties: failureMode: type: string count: type: number additionalProperties: false required: - failureMode - count cases: type: array items: type: object properties: synthetic: type: boolean enum: - true synthesizedAt: type: string sourceCaseIds: type: array items: type: string dimensions: type: object additionalProperties: type: string caseId: type: string input: type: string expected: type: string actual: type: string label: oneOf: - type: string enum: - pass - type: string enum: - fail failureMode: oneOf: - type: 'null' - type: string labeledAt: type: string metadata: type: object properties: behavior: type: string difficulty: type: string tags: type: array items: type: string sourceRunId: oneOf: - type: 'null' - type: number sourceTestCaseId: oneOf: - type: 'null' - type: number agentWorkflow: type: object properties: version: type: number enum: - 1 workflowClass: oneOf: - type: string enum: - approval_gated_mutation - type: string enum: - background_execution - type: string enum: - recovery_required - type: string enum: - policy_conflict - type: string enum: - handoff_heavy - type: string enum: - memory_sensitive - type: string enum: - parallel_tool_use autonomyLevel: oneOf: - type: string enum: - manual - type: string enum: - default_approvals - type: string enum: - bypass_approvals - type: string enum: - autopilot approvalMode: oneOf: - type: string enum: - none - type: string enum: - human_review - type: string enum: - policy_guided - type: string enum: - auto_with_audit riskLevel: oneOf: - type: string enum: - medium - type: string enum: - high - type: string enum: - critical - type: string enum: - low memoryMode: oneOf: - type: string enum: - none - type: string enum: - session - type: string enum: - mixed - type: string enum: - persistent backgroundExecution: type: boolean requiresCheckpoint: type: boolean expectedHandoff: type: boolean requiredHooks: type: array items: oneOf: - type: string enum: - session_start - type: string enum: - pre_plan - type: string enum: - pre_tool - type: string enum: - post_tool - type: string enum: - pre_write - type: string enum: - pre_handoff - type: string enum: - on_error - type: string enum: - session_end activeTools: type: array items: type: string permissionScope: type: object properties: filesystem: type: array items: type: string network: oneOf: - type: string enum: - open - type: string enum: - none - type: string enum: - restricted externalSystems: type: array items: type: string mutatesState: oneOf: - type: boolean enum: - false - type: boolean enum: - true additionalProperties: false expectedFailureModes: type: array items: type: string scenario: type: string provenance: type: object properties: source: oneOf: - type: string enum: - evaluation_run - type: string enum: - manual - type: string enum: - coverage_gap - type: string enum: - trace_failure evaluationId: type: number runId: type: number testCaseId: type: number traceId: oneOf: - type: 'null' - type: string additionalProperties: false required: - source additionalProperties: false required: - version - workflowClass - autonomyLevel - approvalMode - riskLevel - memoryMode - backgroundExecution - requiresCheckpoint - expectedHandoff - requiredHooks - activeTools - permissionScope - expectedFailureModes - scenario workflowExperimentRecipe: oneOf: - type: 'null' - type: object properties: version: type: number enum: - 1 title: type: string objective: type: string workflowClass: oneOf: - type: string enum: - approval_gated_mutation - type: string enum: - background_execution - type: string enum: - recovery_required - type: string enum: - policy_conflict - type: string enum: - handoff_heavy - type: string enum: - memory_sensitive - type: string enum: - parallel_tool_use source: oneOf: - type: string enum: - coverage_gap - type: string enum: - failure linkedFailureModes: type: array items: type: string linkedTestCaseIds: type: array items: type: number comparedVariants: type: array items: type: object properties: id: type: string label: type: string changes: type: array items: type: string autonomyLevel: oneOf: - type: string enum: - manual - type: string enum: - default_approvals - type: string enum: - bypass_approvals - type: string enum: - autopilot approvalMode: oneOf: - type: string enum: - none - type: string enum: - human_review - type: string enum: - policy_guided - type: string enum: - auto_with_audit memoryMode: oneOf: - type: string enum: - none - type: string enum: - session - type: string enum: - mixed - type: string enum: - persistent hooks: type: array items: oneOf: - type: string enum: - session_start - type: string enum: - pre_plan - type: string enum: - pre_tool - type: string enum: - post_tool - type: string enum: - pre_write - type: string enum: - pre_handoff - type: string enum: - on_error - type: string enum: - session_end expectedImpact: type: string additionalProperties: false required: - id - label - changes - expectedImpact successMetrics: type: array items: type: string guardrails: type: array items: type: string recommendedIterations: type: number rationale: type: string additionalProperties: false required: - version - title - objective - workflowClass - source - linkedFailureModes - linkedTestCaseIds - comparedVariants - successMetrics - guardrails - recommendedIterations - rationale additionalProperties: false additionalProperties: false required: - synthetic - synthesizedAt - sourceCaseIds - dimensions - caseId - input - expected - actual - label - failureMode - labeledAt additionalProperties: false required: - missionId - proposalsInserted - continuation - sourceCases - sourceFailures - selectedFailureModes - dimensionNames - dimensionCombinationCount - generated - modeCounts - cases PostApiEvalgateSynthesizeResponse201ApplicationJson: oneOf: - type: 'null' - type: object properties: items: type: array items: type: object properties: id: type: string organizationId: $ref: '#/components/schemas/OrganizationId' generationRunId: type: string stableItemId: type: string ordinal: type: number status: type: string input: type: string expectedOutput: oneOf: - type: 'null' - type: string actual: oneOf: - type: 'null' - type: string failureMode: oneOf: - type: 'null' - type: string dimensions: type: object additionalProperties: type: string sourceCaseIds: type: array items: type: string dedupeHash: type: string provenance: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' validation: oneOf: - type: 'null' - type: object properties: status: oneOf: - type: string enum: - unknown - type: string enum: - pass - type: string enum: - fail issues: type: array items: type: object properties: path: type: string message: type: string additionalProperties: false required: - path - message checks: type: object additionalProperties: oneOf: - type: string enum: - unknown - type: string enum: - pass - type: string enum: - fail validatedAt: type: string additionalProperties: false required: - status - issues - checks - validatedAt execution: oneOf: - type: 'null' - type: object properties: status: oneOf: - type: string enum: - pending - type: string enum: - pass - type: string enum: - fail - type: string enum: - skipped output: oneOf: - type: 'null' - type: string score: oneOf: - type: 'null' - type: number error: oneOf: - type: 'null' - type: string latencyMs: oneOf: - type: 'null' - type: number model: oneOf: - type: 'null' - type: string provider: oneOf: - type: 'null' - type: string modelCallId: oneOf: - type: 'null' - type: string costUsd: oneOf: - type: 'null' - type: number tokens: oneOf: - type: 'null' - type: object properties: input: type: number output: type: number additionalProperties: false required: - input - output checkedAt: type: string additionalProperties: false required: - status - checkedAt attempt: type: number reviewDecision: oneOf: - type: 'null' - type: string reviewRationale: oneOf: - type: 'null' - type: string reviewerId: oneOf: - type: 'null' - type: string reviewedAt: oneOf: - type: 'null' - type: string format: date-time errorMessage: oneOf: - type: 'null' - type: string createdAt: type: string format: date-time updatedAt: type: string format: date-time additionalProperties: false required: - id - organizationId - generationRunId - stableItemId - ordinal - status - input - expectedOutput - actual - failureMode - dimensions - sourceCaseIds - dedupeHash - provenance - validation - execution - attempt - reviewDecision - reviewRationale - reviewerId - reviewedAt - errorMessage - createdAt - updatedAt nextItemCursor: oneOf: - type: 'null' - type: string recipe: type: object properties: source: type: object properties: type: oneOf: - type: string enum: - dataset - type: string enum: - evaluation_run - type: string enum: - trace - type: string enum: - dataset_content - type: string enum: - failure_cluster evaluationRunId: type: number datasetId: type: string traceIds: type: array items: type: number failureClusterIds: type: array items: type: string additionalProperties: false required: - type schema: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' distributionTargets: type: object additionalProperties: type: object additionalProperties: type: number dimensions: type: object additionalProperties: type: array items: type: string failureModes: type: array items: type: string count: type: number maxCombinations: type: number seed: type: string evaluationId: type: number datasetId: type: string stageMissionProposals: type: boolean publishDatasetVersion: type: boolean modelPolicy: type: object properties: mode: oneOf: - type: string enum: - deterministic - type: string enum: - llm provider: type: string model: type: string systemPrompt: type: string temperature: type: number maxTokens: type: number additionalProperties: false required: - mode budget: type: object properties: maxTokens: type: number maxUsd: type: number maxItems: type: number additionalProperties: false execution: type: object properties: enabled: type: boolean required: type: boolean minimumScore: type: number timeoutMs: type: number additionalProperties: false required: - enabled - required - minimumScore - timeoutMs additionalProperties: false required: - source - dimensions - failureModes - count - maxCombinations - seed - stageMissionProposals - publishDatasetVersion - modelPolicy - budget - execution id: type: string organizationId: $ref: '#/components/schemas/OrganizationId' evaluationId: oneOf: - type: 'null' - type: number datasetId: oneOf: - type: 'null' - type: string createdBy: oneOf: - type: 'null' - type: string idempotencyKey: type: string recipeHash: type: string sourceSnapshotHash: type: string sourceRefs: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' modelPolicy: type: object properties: mode: oneOf: - type: string enum: - deterministic - type: string enum: - llm provider: type: string model: type: string systemPrompt: type: string temperature: type: number maxTokens: type: number additionalProperties: false required: - mode budgetPolicy: type: object properties: maxTokens: type: number maxUsd: type: number maxItems: type: number additionalProperties: false lineage: type: object properties: sourceRefs: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' modelCalls: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' provider: oneOf: - type: 'null' - type: string model: oneOf: - type: 'null' - type: string configHash: oneOf: - type: 'null' - type: string budgetReservedUsd: type: number budgetConsumedUsd: type: number budgetConsumedTokens: type: number additionalProperties: false required: - sourceRefs - modelCalls qualitySummary: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' consumedCostUsd: type: string consumedTokens: type: number status: type: string totalCount: type: number pendingCount: type: number validCount: type: number invalidCount: type: number duplicateCount: type: number reviewedCount: type: number missionId: oneOf: - type: 'null' - type: string publishedDatasetVersionId: oneOf: - type: 'null' - type: string errorMessage: oneOf: - type: 'null' - type: string createdAt: type: string format: date-time updatedAt: type: string format: date-time completedAt: oneOf: - type: 'null' - type: string format: date-time additionalProperties: false required: - recipe - id - organizationId - evaluationId - datasetId - createdBy - idempotencyKey - recipeHash - sourceSnapshotHash - sourceRefs - modelPolicy - budgetPolicy - lineage - qualitySummary - consumedCostUsd - consumedTokens - status - totalCount - pendingCount - validCount - invalidCount - duplicateCount - reviewedCount - missionId - publishedDatasetVersionId - errorMessage - createdAt - updatedAt - completedAt PostApiEvalgateSynthesizeResponse202ApplicationJson: type: object properties: continuation: type: object properties: entryPoint: oneOf: - type: string enum: - repository - type: string enum: - playground - type: string enum: - mcp - type: string enum: - trace - type: string enum: - synthetic - type: string enum: - application-health - type: string enum: - auto - type: string enum: - harness workspace: oneOf: - type: string enum: - local - type: string enum: - evidence - type: string enum: - improve - type: string enum: - connect steps: type: array items: type: object properties: href: type: string id: type: string label: type: string description: type: string command: type: string additionalProperties: false required: - href - id - label - description additionalProperties: false required: - entryPoint - workspace - steps items: type: array items: type: object properties: id: type: string organizationId: $ref: '#/components/schemas/OrganizationId' generationRunId: type: string stableItemId: type: string ordinal: type: number status: type: string input: type: string expectedOutput: oneOf: - type: 'null' - type: string actual: oneOf: - type: 'null' - type: string failureMode: oneOf: - type: 'null' - type: string dimensions: type: object additionalProperties: type: string sourceCaseIds: type: array items: type: string dedupeHash: type: string provenance: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' validation: oneOf: - type: 'null' - type: object properties: status: oneOf: - type: string enum: - unknown - type: string enum: - pass - type: string enum: - fail issues: type: array items: type: object properties: path: type: string message: type: string additionalProperties: false required: - path - message checks: type: object additionalProperties: oneOf: - type: string enum: - unknown - type: string enum: - pass - type: string enum: - fail validatedAt: type: string additionalProperties: false required: - status - issues - checks - validatedAt execution: oneOf: - type: 'null' - type: object properties: status: oneOf: - type: string enum: - pending - type: string enum: - pass - type: string enum: - fail - type: string enum: - skipped output: oneOf: - type: 'null' - type: string score: oneOf: - type: 'null' - type: number error: oneOf: - type: 'null' - type: string latencyMs: oneOf: - type: 'null' - type: number model: oneOf: - type: 'null' - type: string provider: oneOf: - type: 'null' - type: string modelCallId: oneOf: - type: 'null' - type: string costUsd: oneOf: - type: 'null' - type: number tokens: oneOf: - type: 'null' - type: object properties: input: type: number output: type: number additionalProperties: false required: - input - output checkedAt: type: string additionalProperties: false required: - status - checkedAt attempt: type: number reviewDecision: oneOf: - type: 'null' - type: string reviewRationale: oneOf: - type: 'null' - type: string reviewerId: oneOf: - type: 'null' - type: string reviewedAt: oneOf: - type: 'null' - type: string format: date-time errorMessage: oneOf: - type: 'null' - type: string createdAt: type: string format: date-time updatedAt: type: string format: date-time additionalProperties: false required: - id - organizationId - generationRunId - stableItemId - ordinal - status - input - expectedOutput - actual - failureMode - dimensions - sourceCaseIds - dedupeHash - provenance - validation - execution - attempt - reviewDecision - reviewRationale - reviewerId - reviewedAt - errorMessage - createdAt - updatedAt nextItemCursor: oneOf: - type: 'null' - type: string recipe: type: object properties: source: type: object properties: type: oneOf: - type: string enum: - dataset - type: string enum: - evaluation_run - type: string enum: - trace - type: string enum: - dataset_content - type: string enum: - failure_cluster evaluationRunId: type: number datasetId: type: string traceIds: type: array items: type: number failureClusterIds: type: array items: type: string additionalProperties: false required: - type schema: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' distributionTargets: type: object additionalProperties: type: object additionalProperties: type: number dimensions: type: object additionalProperties: type: array items: type: string failureModes: type: array items: type: string count: type: number maxCombinations: type: number seed: type: string evaluationId: type: number datasetId: type: string stageMissionProposals: type: boolean publishDatasetVersion: type: boolean modelPolicy: type: object properties: mode: oneOf: - type: string enum: - deterministic - type: string enum: - llm provider: type: string model: type: string systemPrompt: type: string temperature: type: number maxTokens: type: number additionalProperties: false required: - mode budget: type: object properties: maxTokens: type: number maxUsd: type: number maxItems: type: number additionalProperties: false execution: type: object properties: enabled: type: boolean required: type: boolean minimumScore: type: number timeoutMs: type: number additionalProperties: false required: - enabled - required - minimumScore - timeoutMs additionalProperties: false required: - source - dimensions - failureModes - count - maxCombinations - seed - stageMissionProposals - publishDatasetVersion - modelPolicy - budget - execution id: type: string organizationId: $ref: '#/components/schemas/OrganizationId' evaluationId: oneOf: - type: 'null' - type: number datasetId: oneOf: - type: 'null' - type: string createdBy: oneOf: - type: 'null' - type: string idempotencyKey: type: string recipeHash: type: string sourceSnapshotHash: type: string sourceRefs: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' modelPolicy: type: object properties: mode: oneOf: - type: string enum: - deterministic - type: string enum: - llm provider: type: string model: type: string systemPrompt: type: string temperature: type: number maxTokens: type: number additionalProperties: false required: - mode budgetPolicy: type: object properties: maxTokens: type: number maxUsd: type: number maxItems: type: number additionalProperties: false lineage: type: object properties: sourceRefs: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' modelCalls: type: array items: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' provider: oneOf: - type: 'null' - type: string model: oneOf: - type: 'null' - type: string configHash: oneOf: - type: 'null' - type: string budgetReservedUsd: type: number budgetConsumedUsd: type: number budgetConsumedTokens: type: number additionalProperties: false required: - sourceRefs - modelCalls qualitySummary: type: object additionalProperties: $ref: '#/components/schemas/JsonValue' consumedCostUsd: type: string consumedTokens: type: number status: type: string totalCount: type: number pendingCount: type: number validCount: type: number invalidCount: type: number duplicateCount: type: number reviewedCount: type: number missionId: oneOf: - type: 'null' - type: string publishedDatasetVersionId: oneOf: - type: 'null' - type: string errorMessage: oneOf: - type: 'null' - type: string createdAt: type: string format: date-time updatedAt: type: string format: date-time completedAt: oneOf: - type: 'null' - type: string format: date-time additionalProperties: false required: - continuation JsonValue: oneOf: - type: 'null' - type: boolean - type: number - type: string - type: array items: $ref: '#/components/schemas/JsonValue' - type: object additionalProperties: $ref: '#/components/schemas/JsonValue' OrganizationId: type: string format: uuid ApiError: type: object properties: error: type: object properties: code: type: string enum: - UNAUTHORIZED - FORBIDDEN - NOT_FOUND - INVALID_REFERENCE - VALIDATION_ERROR - RATE_LIMITED - CONFLICT - INTERNAL_ERROR - SERVICE_UNAVAILABLE - QUOTA_EXCEEDED - FEATURE_LIMIT_REACHED - NO_ORG_MEMBERSHIP - SHARE_REVOKED - SHARE_EXPIRED - SHARE_UNAVAILABLE - PROVIDER_KEY_MISSING - PROVIDER_KEY_EXPIRED - PROVIDER_RATE_LIMITED - PROVIDER_UNAUTHORIZED - PROVIDER_UNAVAILABLE - AGENT_HANDOFF_INVALID - AGENT_HANDOFF_EXPIRED - AGENT_HANDOFF_DENIED - AGENT_HANDOFF_CONSUMED - PAYLOAD_TOO_LARGE - METHOD_NOT_ALLOWED message: type: string details: {} requestId: type: string format: uuid redaction: type: object properties: applied: type: boolean enum: - true fieldCount: type: integer minimum: 0 secretCount: type: integer minimum: 0 required: - applied - fieldCount - secretCount required: - code - message required: - error responses: ValidationError: description: Validation error content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Unauthorized: description: Unauthorized content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string WWW-Authenticate: description: >- Bearer challenge. Agent-facing routes include the RFC 9728 resource_metadata URL. schema: type: string Forbidden: description: Forbidden content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string NotFound: description: Resource not found content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Conflict: description: Conflict content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string RateLimited: description: Rate limit exceeded content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string Retry-After: description: Delay in seconds before retrying when present. schema: type: integer minimum: 1 InternalError: description: Internal server error content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string ServiceUnavailable: description: Service unavailable content: application/json: schema: $ref: '#/components/schemas/ApiError' headers: RateLimit-Limit: description: Quota in requests for the active window. schema: type: integer minimum: 0 RateLimit-Remaining: description: Remaining requests in the active window. schema: type: integer minimum: 0 RateLimit-Reset: description: Reset delay in seconds (not a Unix timestamp). schema: type: integer minimum: 1 X-RateLimit-Limit: schema: type: integer minimum: 0 X-RateLimit-Remaining: schema: type: integer minimum: 0 X-RateLimit-Reset: description: Legacy reset timestamp in Unix milliseconds. schema: type: integer minimum: 0 RateLimit-Policy: description: IETF HTTPAPI structured quota policy (work in progress). schema: type: string RateLimit: description: >- IETF HTTPAPI structured remaining quota and reset delay (work in progress). schema: type: string securitySchemes: bearerAuth: type: http scheme: bearer bearerFormat: EvalGate API key description: >- Organization-scoped EvalGate API key. Discover the supported agent scopes at /.well-known/oauth-protected-resource. Per-operation grants are listed in x-evalgate-required-scopes; OpenAPI bearer security requirement arrays remain empty as required for non-OAuth schemes. x-evalgate-scopes-supported: - eval:read - eval:write - objectives:read - objectives:write - runs:read - runs:write - traces:read - traces:write - exports:download - reports:write - prompt:publish - scorer:publish - red_team:read - red_team:run - red_team:approve - docs:read - agent:read - agent:execute - resources:read - resources:write - contexts:read - contexts:write - observations:write - checks:read - checks:run - standards:read - standards:write - standards:approve - grants:read - grants:write - admin:keys - admin:workspace - admin:org x-evalgate-protected-resource-metadata: https://www.evalgate.com/.well-known/oauth-protected-resource ```` This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.