> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Llm judge

# LLM judge orchestration in EvalGate

> How EvalGate's judge system works: registry-backed model selection, multi-judge composition, credibility metrics, and enterprise policy enforcement.

An LLM judge is an evaluation backed by a language model that produces a structured result — a score, a pass/fail verdict, and (for generative judges) reasoning tied to specific signals — rather than a raw string response. EvalGate's judge system is not a simple model switcher. It is a control plane: registry-backed model selection, saved presets, deterministic aggregation across multiple judges, per-judge evidence, disagreement handling, and enterprise policy enforcement.

## Judge families

EvalGate uses three related measurement families. Do not collapse them into one "LLM judge" story:

| Family | What it produces | Typical use |
| - | - | - |
| **Rule judges** | Exact, deterministic checks (schema, regex, policy predicates) | Hard contracts that must not depend on model sampling |
| **Generative LLM judges** | Score or pass/fail plus natural-language rationale and signals | Rubric-backed language quality when calibrated against labeled data |
| **Decision judges (Jev)** | Typed Choice, Score, or Noul observations with disposition rules | Fixed-outcome decisions where rationale is not the product |

For typed decision-judge contracts, dispositions, and unresolved states, see [Decision judges: Jev and typed outcomes](https://evalgate.com/docs/platform/decision-judges.md). For how application evaluation, evaluator measurement, qualification, and authorization relate, see [How the evaluation system fits together](https://evalgate.com/docs/concepts/evaluation-layers.md).

## BYOK requirement

LLM judges use **bring your own provider key (BYOK)**. EvalGate does not include
model inference credits: your organization connects a provider or gateway
credential, and that provider bills the judge calls directly. The EvalGate API
key authenticates your request to EvalGate; it does not authorize the model
call.

Before testing a judge, follow [Model providers and BYOK](https://evalgate.com/docs/platform/model-providers-byok.md)
to connect the credential, verify provider health and model availability, and
set provider-side and EvalGate budget controls.

## Current judge guidance

Guidance for **generative** score-model judges (calibration framing; not a universal qualification bar for every EvalGate judge family):

* Prefer deterministic rules for exact checks before asking a model to judge language.
* Use score-model judges only with explicit rubrics, bounded numeric ranges, pass thresholds, and fixed sampling settings.
* Calibrate each judge prompt against human-labeled examples before trusting it in a release gate. Track true positive rate, true negative rate, parse failures, and disagreement. Reliability tiers and agreement rates do not by themselves prove independent task correctness.
* Include few-shot examples when the rubric is nuanced, and keep those examples versioned with the judge.
* Use pairwise judging when "which answer is better" is more reliable than assigning an absolute score.
* Escalate high-disagreement, high-cost, safety-critical, or policy-sensitive cases to human review.
* Store the exact prompt, model, parser, output schema, raw result, cost, latency, and threshold used for every score.
* Keep the model and sampling settings fixed to make comparisons interpretable. Measure repeatability explicitly; fixed settings do not by themselves demonstrate identical outputs or task accuracy.

OpenAI's current grader taxonomy includes string checks, text similarity, score-model graders, and Python code execution. LangSmith's evaluator model separates human, code, LLM-as-judge, and pairwise evaluation techniques. Phoenix recommends validating custom LLM judges against benchmark datasets before relying on them for subjective quality.

Sources: [OpenAI graders](https://platform.openai.com/docs/guides/graders), [LangSmith evaluation concepts](https://docs.langchain.com/langsmith/evaluation-concepts), and [Phoenix custom LLM evaluator benchmark workflow](https://arize.com/docs/phoenix/cookbook/evaluation/creating-a-custom-llm-evaluator-with-a-benchmark-dataset).

## Calibration runbook

Do this before letting a judge block merges:

**1. Create a labeled calibration set**

Start with at least 30 human-labeled examples per critical task or failure mode. Include passing cases, obvious failures, borderline cases, and examples where earlier judges disagreed.

**2. Lock the judge artifact**

Version the rubric, prompt, model, parser, threshold, few-shot examples, sampling settings, and expected JSON schema. A judge config change is a measurement change, not a routine code refactor.

**3. Measure reliability**

Run the judge against the labeled set and inspect true positive rate, true negative rate, parse failure rate, confidence interval, disagreement rate, latency, and cost.

**4. Set gate policy**

Use deterministic assertions for hard contracts, then gate judge-backed checks only when reliability clears your configured threshold. Route high-disagreement or low-confidence cases to human review.

**5. Recalibrate on drift**

Re-run calibration after changing judge model, rubric, threshold, dataset slice, or application behavior. EvalGate marks incompatible judge configurations so trend lines do not imply false continuity.

```json evalgate.config.json theme={null} theme={null}
{
  "judge": {
    "bootstrapSeed": 42,
    "alignmentThresholds": {
      "tprMin": 0.70,
      "tnrMin": 0.70,
      "minLabeledSamples": 30
    }
  }
}
```

> [!WARNING]
>
> Do not use a fresh LLM judge as the only release gate. Use it as an evidence source until calibration proves it can separate passing and failing cases for your task.

## The registry and presets

Instead of hardcoding provider names and model versions throughout your evaluation code, EvalGate maintains a **registry** of available judges and a library of **presets** that group judge configurations for common use cases.

The registry tracks each judge's reliability tier (`trusted`, `stable`, or `experimental`), availability, and whether it is allowed under your organization's policy. Presets let you pick a named configuration — `default-single`, `balanced-committee`, or `trusted-escalation` — and let EvalGate resolve the right models from the judges your organization allows, without scattering model names across your codebase. A preset is listed only when every judge it needs is available to your organization (your org's default preset is always listed).

```bash theme={null} theme={null}
# List available judges in the registry
npx @evalgate/sdk judge registry

# List built-in presets available to your organization
npx @evalgate/sdk judge presets
```

```ts TypeScript theme={null} theme={null}
import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

const registry = await client.llmJudge.listRegistry();
const presets = await client.llmJudge.listPresets();
```

```python Python theme={null} theme={null}
from evalgate_sdk import AIEvalClient

client = AIEvalClient(api_key="sk-...")

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()
```

> [!TIP]
>
> Start with a preset. Presets pin judges and sampling settings so comparisons stay interpretable. Measure repeatability on labeled cases; locked temperature alone does not prove identical outputs. Use the registry to explore alternatives once you have a baseline.

## Testing a judge configuration

Before using a judge in production evaluations, test its behavior on a representative input with `testConfig`. This runs the judge against a real input/output pair without saving the configuration and returns `promptPreview` plus a `judgement` with the score, reasoning, signals, and per-judge results.

```ts TypeScript theme={null} theme={null}
import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

const result = await client.llmJudge.testConfig({
  provider: 'openai',
  model: 'gpt-5.5',
  promptTemplate:
    'Return strict JSON with score, passed, reasoning, and signals.',
  judges: [
    {
      id: 'primary',
      type: 'llm',
      provider: 'openai',
      model: 'gpt-5.5',
    },
    {
      id: 'fallback',
      type: 'llm',
      provider: 'anthropic',
      model: 'claude-opus-4-7',
    },
  ],
  aggregation: 'weighted',
  input: 'Cancel my subscription',
  output: "I've canceled your plan effective today.",
  behavior: 'tool_use',
  taskType: 'support',
});

console.log(result.judgement.score, result.judgement.reasoning);
```

```python Python theme={null} theme={null}
from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import TestLLMJudgeConfigParams

client = AIEvalClient(api_key="sk-...")

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.5",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.5",
            },
            {
                "id": "fallback",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-opus-4-7",
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.judgement.score, result.judgement.reasoning)
```

You can also test a judge from the CLI with fine-grained control over provider, model, aggregation strategy, and prompt template:

```bash theme={null} theme={null}
npx @evalgate/sdk judge test \
  --provider openai \
  --model gpt-5.5 \
  --judge openai:gpt-5.5 \
  --judge anthropic:claude-opus-4-7 \
  --aggregation weighted \
  --prompt-template "Score the output for correctness and completeness using structured JSON." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."
```

## Multi-judge composition

A single judge is sufficient for most evaluations. Add more judges when you need higher reliability, fallback coverage, or explicit disagreement analysis.

When you compose multiple judges, you choose an **aggregation strategy** that determines how individual judge verdicts combine into a final result:

| Strategy | Behavior |
| - | - |
| `all_pass` | The case passes only when every judge passes it |
| `any_pass` | The case passes when at least one judge passes it |
| `weighted` | Scores are combined using per-judge weights |
| `primary_fallback` | Use the primary judge result; fall back to the next if the primary fails to produce a valid result |
| `escalate_on_disagreement` | Flag the case for human review when judges disagree instead of resolving the conflict automatically |
| `confidence_escalate` | Decision-judge certainty bands (`confidencePolicy`); uncertain cases go to a frontier judge or human review |

> [!NOTE]
>
> Prefer `weighted` or `escalate_on_disagreement` over averaging. Averaging hides disagreement — `escalate_on_disagreement` surfaces it as a signal that the case may need human review or a clearer rubric.

### A practical progression

**1. Start with one trusted judge**

Choose a preset from the judge workspace or CLI. Keep the model and sampling settings fixed so comparisons are interpretable, then measure repeatability. Inspect case-level results before adding more judges.

**2. Add a rule judge for exact checks**

For schema-sensitive or safety-critical assertions, add a rule-based judge alongside your LLM judge. Rule judges are fast, cheap, and deterministic.

**3. Add a second LLM judge only when needed**

Add a second LLM judge when you have evidence of instability or when disagreement analysis is valuable — not preemptively. Use `escalate_on_disagreement` to make disagreements visible.

## What a judge result exposes

Each judge result in EvalGate carries more than a score. The full result includes:

| Field | Description |
| - | - |
| `score` | Normalized score from 0 to 1 |
| `passed` | Boolean pass/fail verdict |
| `reasoning` | Generative rationale; decision judges return typed `decisions` instead |
| `signals` | Specific behaviors or properties the judge detected |
| `provider`, `model` | Which provider and model produced the result |
| `modelProvenance` | Temperature, top\_p, seed, max tokens, and prompt hash used for the call |
| `latencyMs` | Time taken for the judge call in milliseconds |
| `tokenUsage` | Input and output token counts for the judge call |
| `retries` | Number of retries needed to get a valid result |
| `parseStatus` | `ok`, `retried`, `failed`, or `not_applicable` (decision judges) |

Multi-judge results also carry `aggregation`, per-judge `judgeResults`, `flags`, `variance`, and a `disagreement` summary (outlier judge IDs and whether the case requires escalation).

This provenance lets you reproduce any score, debug unexpected verdicts, and audit judge behavior over time.

## Judge credibility

A judge that produces unreliable scores is worse than no judge — it gives you false confidence. EvalGate tracks credibility metrics for every configured judge:

> [accordion] True positive rate (TPR) and true negative rate (TNR)
>
> TPR measures how often the judge correctly identifies failures. TNR measures how often it correctly identifies passing cases. Both are gated in `evalgate.config.json` (or with `--judgeTprMin`, `--judgeTnrMin`, and `--judgeMinLabeledSamples` on `evalgate check`):
>
> ```json theme={null} theme={null}
> {
>   "judge": {
>     "bootstrapSeed": 42,
>     "alignmentThresholds": {
>       "tprMin": 0.70,
>       "tnrMin": 0.70,
>       "minLabeledSamples": 30
>     }
>   }
> }
> ```
>
> When discriminative power (TPR + TNR − 1) falls to 0.05 or below, EvalGate skips score correction and reports the raw pass rate with `judge_too_weak_to_correct`. If judge thresholds are configured, `evalgate check` fails with exit code `10` (judge credibility untrustworthy) instead of silently using a biased score.

> [accordion] Bootstrap confidence intervals
>
> EvalGate computes 95% bootstrap confidence intervals on corrected pass rates. With fewer than 30 labeled samples, CI computation is skipped and the `judgeCredibility` block of the JSON report records `ciSkippedReason: "insufficient_samples_for_ci"`.
>
> Set `bootstrapSeed` in your config for deterministic CI runs.

> [accordion] Reliability tiers
>
> Each generative judge in the registry is assigned a reliability tier from your organization's recorded judge results:
>
> | Tier | Criteria |
> | - | - |
> | `experimental` | Preview model, or parse/malformed rate > 8%, or variance > 0.2 |
> | `trusted` | Recommended model, or parse failures ≤ 3%, retries ≤ 5%, disagreement ≤ 20% |
> | `stable` | Everything else |
>
> Decision judges are listed as `experimental` until calibrated. Availability is separate from tier: a judge is disabled when your org policy does not allow its provider or model.

> [accordion] Disagreement analysis
>
> In multi-judge runs, EvalGate measures score spread (standard deviation, range, min/max), pass/fail splits across judges, and outlier judges deviating more than 0.3 from the group mean. Cases with high disagreement (`range ≥ 0.4` or any pass/fail split) are flagged for human review — they are often the most informative cases in your dataset.

## Enterprise controls

Before EvalGate sends any data to an external judge provider, it enforces your organization's policy rules:

* **Provider and model allowlists** — External providers can be disabled outright; only allowlisted providers and models are available in the registry and can receive evaluation data
* **Sensitive-data egress checks** — Judge inputs are classified, and calls carrying detected PII are blocked unless your PII policy allows external transfer
* **Egress URL allowlist** — Provider calls to non-allowlisted endpoints are blocked in strict mode
* **Cost and latency ceilings** — Org policy sets a maximum run cost and judge latency; built-in presets are capped to those values
* **Multi-judge policy** — When multi-judge is disabled, built-in presets fall back to a single judge
* **Audit logging** — Blocked provider calls and judge configuration changes are written to the audit log, and judge results link to the model call ledger

> [!WARNING]
>
> A judge that receives un-redacted PII produces results that are not just incorrect but potentially harmful from a compliance standpoint. Review your org's PII policy before configuring external judge providers.

## Where judges appear in the product

The UI labels LLM judges as **AI reviewers**.

* **Measurement → Judges** (`/measurement?view=judges`) — Judge configurations, composition and aggregation, execution envelope, cost vs. quality, a test playground, and registry reliability tiers
* **Measurement → Calibration** — How closely a judge tracks human labels
* **Review queue** (`/review`) — Cases routed for human review, including judge disagreement
* **Experiment comparison** — Filter compared rows by disagreement, regression, cost, and tool errors

## Related guides

* [Decision judges: Jev and typed outcomes](https://evalgate.com/docs/platform/decision-judges.md) — Choice, Score, and Noul observations versus generative rationale
* [How the evaluation system fits together](https://evalgate.com/docs/concepts/evaluation-layers.md) — Application behavior, evaluator measurement, qualification, and authorization
* [Read evaluation evidence](https://evalgate.com/docs/guides/read-evaluation-evidence.md) — Observation versus terminal outcome and unresolved states


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.