LLM judge orchestration in EvalGate

How EvalGate’s judge system works: registry-backed model selection, multi-judge composition, credibility metrics, and enterprise policy enforcement.

An LLM judge is an evaluation backed by a language model that produces a structured result — a score, a pass/fail verdict, and (for generative judges) reasoning tied to specific signals — rather than a raw string response. EvalGate’s judge system is not a simple model switcher. It is a control plane: registry-backed model selection, saved presets, deterministic aggregation across multiple judges, per-judge evidence, disagreement handling, and enterprise policy enforcement.

Judge families

EvalGate uses three related measurement families. Do not collapse them into one “LLM judge” story:

Family	What it produces	Typical use
Rule judges	Exact, deterministic checks (schema, regex, policy predicates)	Hard contracts that must not depend on model sampling
Generative LLM judges	Score or pass/fail plus natural-language rationale and signals	Rubric-backed language quality when calibrated against labeled data
Decision judges (Jev)	Typed Choice, Score, or Noul observations with disposition rules	Fixed-outcome decisions where rationale is not the product

For typed decision-judge contracts, dispositions, and unresolved states, see Decision judges: Jev and typed outcomes. For how application evaluation, evaluator measurement, qualification, and authorization relate, see How the evaluation system fits together.

BYOK requirement

LLM judges use bring your own provider key (BYOK). EvalGate does not include model inference credits: your organization connects a provider or gateway credential, and that provider bills the judge calls directly. The EvalGate API key authenticates your request to EvalGate; it does not authorize the model call. Before testing a judge, follow Model providers and BYOK to connect the credential, verify provider health and model availability, and set provider-side and EvalGate budget controls.

Current judge guidance

Guidance for generative score-model judges (calibration framing; not a universal qualification bar for every EvalGate judge family):

Prefer deterministic rules for exact checks before asking a model to judge language.
Use score-model judges only with explicit rubrics, bounded numeric ranges, pass thresholds, and fixed sampling settings.
Calibrate each judge prompt against human-labeled examples before trusting it in a release gate. Track true positive rate, true negative rate, parse failures, and disagreement. Reliability tiers and agreement rates do not by themselves prove independent task correctness.
Include few-shot examples when the rubric is nuanced, and keep those examples versioned with the judge.
Use pairwise judging when “which answer is better” is more reliable than assigning an absolute score.
Escalate high-disagreement, high-cost, safety-critical, or policy-sensitive cases to human review.
Store the exact prompt, model, parser, output schema, raw result, cost, latency, and threshold used for every score.
Keep the model and sampling settings fixed to make comparisons interpretable. Measure repeatability explicitly; fixed settings do not by themselves demonstrate identical outputs or task accuracy.

OpenAI’s current grader taxonomy includes string checks, text similarity, score-model graders, and Python code execution. LangSmith’s evaluator model separates human, code, LLM-as-judge, and pairwise evaluation techniques. Phoenix recommends validating custom LLM judges against benchmark datasets before relying on them for subjective quality. Sources: OpenAI graders, LangSmith evaluation concepts, and Phoenix custom LLM evaluator benchmark workflow.

Calibration runbook

Do this before letting a judge block merges:

Create a labeled calibration set

Start with at least 30 human-labeled examples per critical task or failure mode. Include passing cases, obvious failures, borderline cases, and examples where earlier judges disagreed.

Lock the judge artifact

Version the rubric, prompt, model, parser, threshold, few-shot examples, sampling settings, and expected JSON schema. A judge config change is a measurement change, not a routine code refactor.

Measure reliability

Run the judge against the labeled set and inspect true positive rate, true negative rate, parse failure rate, confidence interval, disagreement rate, latency, and cost.

Set gate policy

Use deterministic assertions for hard contracts, then gate judge-backed checks only when reliability clears your configured threshold. Route high-disagreement or low-confidence cases to human review.

Recalibrate on drift

Re-run calibration after changing judge model, rubric, threshold, dataset slice, or application behavior. EvalGate marks incompatible judge configurations so trend lines do not imply false continuity.

evalgate.config.json

{
  "judge": {
    "bootstrapSeed": 42,
    "alignmentThresholds": {
      "tprMin": 0.70,
      "tnrMin": 0.70,
      "minLabeledSamples": 30
    }
  }
}

Do not use a fresh LLM judge as the only release gate. Use it as an evidence source until calibration proves it can separate passing and failing cases for your task.

The registry and presets

Instead of hardcoding provider names and model versions throughout your evaluation code, EvalGate maintains a registry of available judges and a library of presets that group judge configurations for common use cases. The registry tracks each judge’s reliability tier (trusted, stable, or experimental), availability, and whether it is allowed under your organization’s policy. Presets let you pick a named configuration — default-single, balanced-committee, or trusted-escalation — and let EvalGate resolve the right models from the judges your organization allows, without scattering model names across your codebase. A preset is listed only when every judge it needs is available to your organization (your org’s default preset is always listed).

# List available judges in the registry
npx @evalgate/sdk judge registry

# List built-in presets available to your organization
npx @evalgate/sdk judge presets

TypeScript

import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

const registry = await client.llmJudge.listRegistry();
const presets = await client.llmJudge.listPresets();

Python

from evalgate_sdk import AIEvalClient

client = AIEvalClient(api_key="sk-...")

registry = await client.llm_judge.list_registry()
presets = await client.llm_judge.list_presets()

Start with a preset. Presets pin judges and sampling settings so comparisons stay interpretable. Measure repeatability on labeled cases; locked temperature alone does not prove identical outputs. Use the registry to explore alternatives once you have a baseline.

Testing a judge configuration

Before using a judge in production evaluations, test its behavior on a representative input with testConfig. This runs the judge against a real input/output pair without saving the configuration and returns promptPreview plus a judgement with the score, reasoning, signals, and per-judge results.

import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init();

const result = await client.llmJudge.testConfig({
  provider: 'openai',
  model: 'gpt-5.5',
  promptTemplate:
    'Return strict JSON with score, passed, reasoning, and signals.',
  judges: [
    {
      id: 'primary',
      type: 'llm',
      provider: 'openai',
      model: 'gpt-5.5',
    },
    {
      id: 'fallback',
      type: 'llm',
      provider: 'anthropic',
      model: 'claude-opus-4-7',
    },
  ],
  aggregation: 'weighted',
  input: 'Cancel my subscription',
  output: "I've canceled your plan effective today.",
  behavior: 'tool_use',
  taskType: 'support',
});

console.log(result.judgement.score, result.judgement.reasoning);

from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import TestLLMJudgeConfigParams

client = AIEvalClient(api_key="sk-...")

result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider="openai",
        model="gpt-5.5",
        prompt_template="Return strict JSON with score, passed, reasoning, and signals.",
        judges=[
            {
                "id": "primary",
                "type": "llm",
                "provider": "openai",
                "model": "gpt-5.5",
            },
            {
                "id": "fallback",
                "type": "llm",
                "provider": "anthropic",
                "model": "claude-opus-4-7",
            },
        ],
        aggregation="weighted",
        input="Cancel my subscription",
        output="I've canceled your plan effective today.",
        behavior="tool_use",
        task_type="support",
    )
)

print(result.judgement.score, result.judgement.reasoning)

You can also test a judge from the CLI with fine-grained control over provider, model, aggregation strategy, and prompt template:

npx @evalgate/sdk judge test \
  --provider openai \
  --model gpt-5.5 \
  --judge openai:gpt-5.5 \
  --judge anthropic:claude-opus-4-7 \
  --aggregation weighted \
  --prompt-template "Score the output for correctness and completeness using structured JSON." \
  --input "Cancel my subscription" \
  --output "I've canceled your plan effective today."

Multi-judge composition

A single judge is sufficient for most evaluations. Add more judges when you need higher reliability, fallback coverage, or explicit disagreement analysis. When you compose multiple judges, you choose an aggregation strategy that determines how individual judge verdicts combine into a final result:

Strategy	Behavior
`all_pass`	The case passes only when every judge passes it
`any_pass`	The case passes when at least one judge passes it
`weighted`	Scores are combined using per-judge weights
`primary_fallback`	Use the primary judge result; fall back to the next if the primary fails to produce a valid result
`escalate_on_disagreement`	Flag the case for human review when judges disagree instead of resolving the conflict automatically
`confidence_escalate`	Decision-judge certainty bands (`confidencePolicy`); uncertain cases go to a frontier judge or human review

Prefer weighted or escalate_on_disagreement over averaging. Averaging hides disagreement — escalate_on_disagreement surfaces it as a signal that the case may need human review or a clearer rubric.

A practical progression

Start with one trusted judge

Choose a preset from the judge workspace or CLI. Keep the model and sampling settings fixed so comparisons are interpretable, then measure repeatability. Inspect case-level results before adding more judges.

Add a rule judge for exact checks

For schema-sensitive or safety-critical assertions, add a rule-based judge alongside your LLM judge. Rule judges are fast, cheap, and deterministic.

Add a second LLM judge only when needed

Add a second LLM judge when you have evidence of instability or when disagreement analysis is valuable — not preemptively. Use escalate_on_disagreement to make disagreements visible.

What a judge result exposes

Each judge result in EvalGate carries more than a score. The full result includes:

Field	Description
`score`	Normalized score from 0 to 1
`passed`	Boolean pass/fail verdict
`reasoning`	Generative rationale; decision judges return typed `decisions` instead
`signals`	Specific behaviors or properties the judge detected
`provider`, `model`	Which provider and model produced the result
`modelProvenance`	Temperature, top_p, seed, max tokens, and prompt hash used for the call
`latencyMs`	Time taken for the judge call in milliseconds
`tokenUsage`	Input and output token counts for the judge call
`retries`	Number of retries needed to get a valid result
`parseStatus`	`ok`, `retried`, `failed`, or `not_applicable` (decision judges)

Multi-judge results also carry aggregation, per-judge judgeResults, flags, variance, and a disagreement summary (outlier judge IDs and whether the case requires escalation). This provenance lets you reproduce any score, debug unexpected verdicts, and audit judge behavior over time.

Judge credibility

A judge that produces unreliable scores is worse than no judge — it gives you false confidence. EvalGate tracks credibility metrics for every configured judge:

True positive rate (TPR) and true negative rate (TNR)

TPR measures how often the judge correctly identifies failures. TNR measures how often it correctly identifies passing cases. Both are gated in evalgate.config.json (or with --judgeTprMin, --judgeTnrMin, and --judgeMinLabeledSamples on evalgate check):

{
  "judge": {
    "bootstrapSeed": 42,
    "alignmentThresholds": {
      "tprMin": 0.70,
      "tnrMin": 0.70,
      "minLabeledSamples": 30
    }
  }
}

When discriminative power (TPR + TNR − 1) falls to 0.05 or below, EvalGate skips score correction and reports the raw pass rate with judge_too_weak_to_correct. If judge thresholds are configured, evalgate check fails with exit code 10 (judge credibility untrustworthy) instead of silently using a biased score.

Bootstrap confidence intervals

EvalGate computes 95% bootstrap confidence intervals on corrected pass rates. With fewer than 30 labeled samples, CI computation is skipped and the judgeCredibility block of the JSON report records ciSkippedReason: "insufficient_samples_for_ci".Set bootstrapSeed in your config for deterministic CI runs.

Reliability tiers

Each generative judge in the registry is assigned a reliability tier from your organization’s recorded judge results:

Tier	Criteria
`experimental`	Preview model, or parse/malformed rate > 8%, or variance > 0.2
`trusted`	Recommended model, or parse failures ≤ 3%, retries ≤ 5%, disagreement ≤ 20%
`stable`	Everything else

Decision judges are listed as experimental until calibrated. Availability is separate from tier: a judge is disabled when your org policy does not allow its provider or model.

Disagreement analysis

In multi-judge runs, EvalGate measures score spread (standard deviation, range, min/max), pass/fail splits across judges, and outlier judges deviating more than 0.3 from the group mean. Cases with high disagreement (range ≥ 0.4 or any pass/fail split) are flagged for human review — they are often the most informative cases in your dataset.

Enterprise controls

Before EvalGate sends any data to an external judge provider, it enforces your organization’s policy rules:

Provider and model allowlists — External providers can be disabled outright; only allowlisted providers and models are available in the registry and can receive evaluation data
Sensitive-data egress checks — Judge inputs are classified, and calls carrying detected PII are blocked unless your PII policy allows external transfer
Egress URL allowlist — Provider calls to non-allowlisted endpoints are blocked in strict mode
Cost and latency ceilings — Org policy sets a maximum run cost and judge latency; built-in presets are capped to those values
Multi-judge policy — When multi-judge is disabled, built-in presets fall back to a single judge
Audit logging — Blocked provider calls and judge configuration changes are written to the audit log, and judge results link to the model call ledger

A judge that receives un-redacted PII produces results that are not just incorrect but potentially harmful from a compliance standpoint. Review your org’s PII policy before configuring external judge providers.

Where judges appear in the product

The UI labels LLM judges as AI reviewers.

Measurement → Judges (/measurement?view=judges) — Judge configurations, composition and aggregation, execution envelope, cost vs. quality, a test playground, and registry reliability tiers
Measurement → Calibration — How closely a judge tracks human labels
Review queue (/review) — Cases routed for human review, including judge disagreement
Experiment comparison — Filter compared rows by disagreement, regression, cost, and tool errors

Decision judges: Jev and typed outcomes — Choice, Score, and Noul observations versus generative rationale
How the evaluation system fits together — Application behavior, evaluator measurement, qualification, and authorization
Read evaluation evidence — Observation versus terminal outcome and unresolved states

Get Started

Core Concepts

Guides

SDK Reference

Platform

Llm judge

LLM judge orchestration in EvalGate

Judge families

BYOK requirement

Current judge guidance

Calibration runbook

The registry and presets

Testing a judge configuration

Multi-judge composition

A practical progression

What a judge result exposes

Judge credibility

Enterprise controls

Where judges appear in the product

​LLM judge orchestration in EvalGate

​Judge families

​BYOK requirement

​Current judge guidance

​Calibration runbook

​The registry and presets

​Testing a judge configuration

​Multi-judge composition

​A practical progression

​What a judge result exposes

​Judge credibility

​Enterprise controls

​Where judges appear in the product

​Related guides

LLM judge orchestration in EvalGate

Judge families

BYOK requirement

Current judge guidance

Calibration runbook

The registry and presets

Testing a judge configuration

Multi-judge composition

A practical progression

What a judge result exposes

Judge credibility

Enterprise controls

Where judges appear in the product

Related guides