> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Llm judge

# LLM Judge API — evaluate and orchestrate judges

> List judge configurations, evaluate LLM outputs, retrieve results, and compare judge scores with human annotations.

The LLM Judge API lets you programmatically run AI-powered quality assessments on any input/output pair. Use it to build automated evaluation pipelines, compare judge scores with human labels, and retrieve detailed scoring breakdowns — including reasoning, signals, and per-judge results.

> [!WARNING]
>
> Judge calls require [bring your own provider key (BYOK)](https://evalgate.com/docs/platform/model-providers-byok.md). Your EvalGate bearer token authenticates this API request, while the provider or gateway credential authorizes the model call. EvalGate does not bundle inference credits; the connected provider bills usage directly.

## GET /api/llm-judge/configs — list judge configurations

Returns the judge configurations available in your organization as a JSON array. Pass `id` to fetch one configuration, or filter with `search` (name) and `model`. Paginate with `limit` (default 50) and `offset`.

```bash theme={null} theme={null}
curl "https://evalgate.com/api/llm-judge/configs?limit=20" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Response

```json theme={null} theme={null}
[
  {
    "id": 7,
    "name": "Support quality committee",
    "model": "gpt-5.5",
    "provider": "openai",
    "promptTemplate": "Score the response for correctness and completeness.",
    "judges": [
      { "id": "primary", "type": "llm", "provider": "openai", "model": "gpt-5.5" }
    ],
    "aggregation": "weighted",
    "passThreshold": 0.7,
    "createdAt": "2026-09-01T09:00:00.000Z"
  }
]
```

The same route supports `POST` (create), `PUT ?id=` (update), and `DELETE ?id=` (delete). To try a draft configuration without saving it, use `POST /api/llm-judge/test`.

***

## POST /api/llm-judge/evaluate — evaluate an output

Submits an input/output pair for evaluation by a saved judge configuration.

```bash theme={null} theme={null}
curl https://evalgate.com/api/llm-judge/evaluate \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "configId": 7,
    "input": "Cancel my subscription",
    "output": "I have canceled your plan effective today. You will retain access through the end of the billing period."
  }'
```

### Request body

**configId** `integer` *required*

ID of a saved judge configuration. To test an unsaved configuration, use `POST /api/llm-judge/test`.

**input** `string` *required*

The original prompt or user query that was sent to your LLM.

**output** `string` *required*

The LLM response to evaluate.

**expectedOutput** `string`

Optional reference answer for the judge to compare against.

**behavior** `string`

Optional behavior label (for example `tool_use`).

**taskType** `string`

Optional task type (for example `support`).

**context** `string | object`

Optional extra context passed to the judge prompt.

**metadata** `object`

Optional metadata stored with the result.

### Response

Returns `201 Created`.

```json theme={null} theme={null}
{
  "result": {
    "judgeId": "aggregate",
    "provider": "aggregate",
    "model": "aggregate",
    "score": 0.92,
    "passed": true,
    "reasoning": "The response correctly fulfills the cancellation request and clearly communicates the effective date and access period.",
    "signals": { "clear_confirmation": true, "billing_period_noted": true },
    "latencyMs": 1320,
    "tokenUsage": { "input": 160, "output": 50, "total": 210 },
    "retries": 0,
    "parseStatus": "ok",
    "aggregation": "weighted",
    "judgeResults": [],
    "flags": [],
    "disagreement": {
      "rate": 0,
      "reasoningDivergence": 0,
      "outlierJudgeIds": [],
      "requiresEscalation": false
    }
  }
}
```

**result.score** `number`

Normalized score from 0 to 1.

**result.passed** `boolean`

Whether the output met the configuration's `passThreshold`.

**result.reasoning** `string`

The judge's natural-language explanation of the score.

**result.signals** `object`

Map of signal names to values the judge identified — positive indicators, failure patterns, or flagged behaviors.

**result.latencyMs** `number`

Time in milliseconds the judge call took.

**result.tokenUsage** `object`

Input, output, and total tokens consumed by the judge call.

**result.retries** `integer`

Number of retries the judge performed before returning a parseable result.

**result.parseStatus** `string`

`ok`, `retried`, `failed`, or `not_applicable` (decision judges).

**result.judgeResults** `array`

Per-judge executions, each with its own result, provider, model, and any failure code.

**result.disagreement** `object`

Disagreement summary across judges: outlier judge IDs and whether the case requires escalation.

If every configured judge fails (for example, all are blocked by policy or rate limited), the API returns an error with `error.details.reason: "JUDGE_EXECUTIONS_FAILED"` and the per-judge failure codes.

***

## GET /api/llm-judge/results — get evaluation results

Returns stored evaluation results as a JSON array for review, filtering, or export.

```bash theme={null} theme={null}
curl "https://evalgate.com/api/llm-judge/results?configId=7&limit=50" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Query parameters

**configId** `integer`

Filter results to a specific judge configuration.

**limit** `integer`

Maximum number of results to return. Defaults to 50.

**offset** `integer`

Pagination offset. Defaults to 0.

***

## GET /api/llm-judge/alignment

Compare judge scores with human annotations for one evaluation run.

For each test case in the run that has both a judge result and a human annotation, EvalGate compares the judge score (0–100) with the human rating (1–5) after normalizing both to 0–1. Collect human labels first with the [Annotations API](https://evalgate.com/docs/api/annotations.md).

```bash theme={null} theme={null}
curl "https://evalgate.com/api/llm-judge/alignment?evaluationRunId=42" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Query parameters

**evaluationRunId** `integer` *required*

ID of the evaluation run whose judge results and human annotations to compare.

### Response

```json theme={null} theme={null}
{
  "alignmentData": [
    {
      "testCaseId": 301,
      "judgeScore": 90,
      "humanRating": 5,
      "alignment": 0.9,
      "judgeReasoning": "Clear confirmation of the cancellation.",
      "humanFeedback": "Correct and complete."
    }
  ],
  "metrics": {
    "averageAlignment": 0.88,
    "totalComparisons": 120,
    "highAlignment": 98,
    "lowAlignment": 4,
    "alignmentRate": 0.82
  }
}
```

**alignmentData\[].alignment** `number`

Per-case agreement: 1 minus the absolute difference between the normalized judge score and human rating.

**metrics.averageAlignment** `number`

Mean per-case agreement across compared cases.

**metrics.highAlignment** `integer`

Cases with agreement of 0.8 or higher.

**metrics.lowAlignment** `integer`

Cases with agreement below 0.5.

**metrics.alignmentRate** `number`

Share of compared cases with high agreement.

> [!TIP]
>
> Before using a judge to gate CI runs, measure its true positive and true negative rates against labeled cases and set `judge.alignmentThresholds` in `evalgate.config.json`. See [Judge credibility](https://evalgate.com/docs/concepts/llm-judge.md#judge-credibility).

## Related endpoints

* `POST /api/llm-judge/test` — run a draft judge configuration without saving it
* `POST /api/llm-judge/compare` — compare two outputs with the same saved configuration
* `GET /api/judges` — list the judge registry for your organization
* `GET /api/judge-presets` — list built-in judge presets available to your organization
* `GET /api/runs/{id}/judges` — list judge results for an evaluation run


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.