LLM Judge API — evaluate and orchestrate judges

List judge configurations, evaluate LLM outputs, retrieve results, and compare judge scores with human annotations.

The LLM Judge API lets you programmatically run AI-powered quality assessments on any input/output pair. Use it to build automated evaluation pipelines, compare judge scores with human labels, and retrieve detailed scoring breakdowns — including reasoning, signals, and per-judge results.

Judge calls require bring your own provider key (BYOK). Your EvalGate bearer token authenticates this API request, while the provider or gateway credential authorizes the model call. EvalGate does not bundle inference credits; the connected provider bills usage directly.

GET /api/llm-judge/configs — list judge configurations

Returns the judge configurations available in your organization as a JSON array. Pass id to fetch one configuration, or filter with search (name) and model. Paginate with limit (default 50) and offset.

curl "https://evalgate.com/api/llm-judge/configs?limit=20" \
  -H "Authorization: Bearer YOUR_API_KEY"

Response

[
  {
    "id": 7,
    "name": "Support quality committee",
    "model": "gpt-5.5",
    "provider": "openai",
    "promptTemplate": "Score the response for correctness and completeness.",
    "judges": [
      { "id": "primary", "type": "llm", "provider": "openai", "model": "gpt-5.5" }
    ],
    "aggregation": "weighted",
    "passThreshold": 0.7,
    "createdAt": "2026-09-01T09:00:00.000Z"
  }
]

The same route supports POST (create), PUT ?id= (update), and DELETE ?id= (delete). To try a draft configuration without saving it, use POST /api/llm-judge/test.

POST /api/llm-judge/evaluate — evaluate an output

Submits an input/output pair for evaluation by a saved judge configuration.

curl https://evalgate.com/api/llm-judge/evaluate \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "configId": 7,
    "input": "Cancel my subscription",
    "output": "I have canceled your plan effective today. You will retain access through the end of the billing period."
  }'

Request body

integer

required

ID of a saved judge configuration. To test an unsaved configuration, use POST /api/llm-judge/test.

string

required

The original prompt or user query that was sent to your LLM.

string

required

The LLM response to evaluate.

string

Optional reference answer for the judge to compare against.

string

Optional behavior label (for example tool_use).

string

Optional task type (for example support).

string | object

Optional extra context passed to the judge prompt.

object

Optional metadata stored with the result.

Response

Returns 201 Created.

{
  "result": {
    "judgeId": "aggregate",
    "provider": "aggregate",
    "model": "aggregate",
    "score": 0.92,
    "passed": true,
    "reasoning": "The response correctly fulfills the cancellation request and clearly communicates the effective date and access period.",
    "signals": { "clear_confirmation": true, "billing_period_noted": true },
    "latencyMs": 1320,
    "tokenUsage": { "input": 160, "output": 50, "total": 210 },
    "retries": 0,
    "parseStatus": "ok",
    "aggregation": "weighted",
    "judgeResults": [],
    "flags": [],
    "disagreement": {
      "rate": 0,
      "reasoningDivergence": 0,
      "outlierJudgeIds": [],
      "requiresEscalation": false
    }
  }
}

number

Normalized score from 0 to 1.

boolean

Whether the output met the configuration’s passThreshold.

string

The judge’s natural-language explanation of the score.

object

Map of signal names to values the judge identified — positive indicators, failure patterns, or flagged behaviors.

number

Time in milliseconds the judge call took.

object

Input, output, and total tokens consumed by the judge call.

integer

Number of retries the judge performed before returning a parseable result.

string

ok, retried, failed, or not_applicable (decision judges).

array

Per-judge executions, each with its own result, provider, model, and any failure code.

object

Disagreement summary across judges: outlier judge IDs and whether the case requires escalation.

If every configured judge fails (for example, all are blocked by policy or rate limited), the API returns an error with error.details.reason: "JUDGE_EXECUTIONS_FAILED" and the per-judge failure codes.

GET /api/llm-judge/results — get evaluation results

Returns stored evaluation results as a JSON array for review, filtering, or export.

curl "https://evalgate.com/api/llm-judge/results?configId=7&limit=50" \
  -H "Authorization: Bearer YOUR_API_KEY"

Query parameters

integer

Filter results to a specific judge configuration.

integer

Maximum number of results to return. Defaults to 50.

integer

Pagination offset. Defaults to 0.

GET /api/llm-judge/alignment

Compare judge scores with human annotations for one evaluation run. For each test case in the run that has both a judge result and a human annotation, EvalGate compares the judge score (0–100) with the human rating (1–5) after normalizing both to 0–1. Collect human labels first with the Annotations API.

curl "https://evalgate.com/api/llm-judge/alignment?evaluationRunId=42" \
  -H "Authorization: Bearer YOUR_API_KEY"

Query parameters

integer

required

ID of the evaluation run whose judge results and human annotations to compare.

Response

{
  "alignmentData": [
    {
      "testCaseId": 301,
      "judgeScore": 90,
      "humanRating": 5,
      "alignment": 0.9,
      "judgeReasoning": "Clear confirmation of the cancellation.",
      "humanFeedback": "Correct and complete."
    }
  ],
  "metrics": {
    "averageAlignment": 0.88,
    "totalComparisons": 120,
    "highAlignment": 98,
    "lowAlignment": 4,
    "alignmentRate": 0.82
  }
}

number

Per-case agreement: 1 minus the absolute difference between the normalized judge score and human rating.

number

Mean per-case agreement across compared cases.

integer

Cases with agreement of 0.8 or higher.

integer

Cases with agreement below 0.5.

number

Share of compared cases with high agreement.

Before using a judge to gate CI runs, measure its true positive and true negative rates against labeled cases and set judge.alignmentThresholds in evalgate.config.json. See Judge credibility.

POST /api/llm-judge/test — run a draft judge configuration without saving it
POST /api/llm-judge/compare — compare two outputs with the same saved configuration
GET /api/judges — list the judge registry for your organization
GET /api/judge-presets — list built-in judge presets available to your organization
GET /api/runs/{id}/judges — list judge results for an evaluation run

Overview

Endpoints

Llm judge

LLM Judge API — evaluate and orchestrate judges

GET /api/llm-judge/configs — list judge configurations

Response

POST /api/llm-judge/evaluate — evaluate an output

Request body

Response

GET /api/llm-judge/results — get evaluation results

Query parameters

GET /api/llm-judge/alignment

Query parameters

Response

​LLM Judge API — evaluate and orchestrate judges

​GET /api/llm-judge/configs — list judge configurations

​Response

​POST /api/llm-judge/evaluate — evaluate an output

​Request body

​Response

​GET /api/llm-judge/results — get evaluation results

​Query parameters

​GET /api/llm-judge/alignment

​Query parameters

​Response

​Related endpoints

LLM Judge API — evaluate and orchestrate judges

GET /api/llm-judge/configs — list judge configurations

Response

POST /api/llm-judge/evaluate — evaluate an output

Request body

Response

GET /api/llm-judge/results — get evaluation results

Query parameters

GET /api/llm-judge/alignment

Query parameters

Response

Related endpoints