Create a scorer

A scorer turns evaluation evidence into a score or decision. Start with the cheapest deterministic check that captures the requirement, and add model-backed judgment only where semantic interpretation earns its cost.

Pick the right kind

Kind	When to use it	Example
Deterministic	Exact, rule-based checks	Output contains a required identifier
Code	Logic too complex for a simple rule	Parse, normalize, and compare structured output
Composite	Combine several scorer signals	Weighted deterministic + judge evidence
LLM judge	Semantic judgment is required	Is the answer safe and on-topic?

Step 1 — Open Measurement → Scorers

Open /measurement?view=scorers and choose New scorer. The old /scorers list route redirects into this Measurement view. Existing scorer detail links such as /scorers/{scorerId} remain valid.

Step 2 — Define the scorer

Choose the scorer kind and configure the behavior it owns. Keep model/provider, prompt, thresholds, mappings, examples, and other decision inputs versioned where the scorer type exposes them. Code-backed execution is governed by the scorer runtime boundary; do not assume your local Node or Python environment is identical to production execution.

Step 3 — Add and review examples

Add labeled examples that represent both acceptable behavior and important failures. Examples are measurement evidence, not a substitute for independent qualification.

Step 4 — Test and qualify

Run the scorer against its examples and inspect disagreements. Where the workflow requires evaluator qualification, complete that evidence before using the scorer as release authority.

Step 5 — Approve, publish, and bind the exact version

Publishing/binding uses immutable scorer versions. A later edit creates new evidence; it does not silently rewrite the version used by an earlier run.

Up next

Calibrate the scorer when scores need cross-version or cross-model comparability.

How to use EvalGate

EvalGate 101

Evaluations & datasets

Repository intelligence

Traces & logs

Proof

Prompts & scorers

Playground & experiments

Continuous eval & monitors

Copilot & review

Deployments & reports

Settings & account

Create a scorer

Create a scorer

Pick the right kind

Step 1 — Open Measurement → Scorers

Step 2 — Define the scorer

Step 3 — Add and review examples

Step 4 — Test and qualify

Step 5 — Approve, publish, and bind the exact version

Up next

​Create a scorer

​Pick the right kind

​Step 1 — Open Measurement → Scorers

​Step 2 — Define the scorer

​Step 3 — Add and review examples

​Step 4 — Test and qualify

​Step 5 — Approve, publish, and bind the exact version

​Up next

Create a scorer

Pick the right kind

Step 1 — Open Measurement → Scorers

Step 2 — Define the scorer

Step 3 — Add and review examples

Step 4 — Test and qualify

Step 5 — Approve, publish, and bind the exact version

Up next