Skip to main content

​
Scorer Studio

Scorer Studio is EvalGate’s organization-scoped registry and evidence workbench for reusable scoring logic. A scorer has a stable key, while every saved definition becomes a new immutable release_artifact version with a numeric versionId, a SHA-256 contentHash, and, for code packages, a signed package hash. Use Scorer Studio when Playground, Experiments, Online Eval, and release gates must execute the same reviewed scoring definition and retain the exact score scale, implementation, prompt/model binding, calibration mapping, and provenance that produced each result.
A consumer binds an exact versionId and contentHash. Scorer Studio never resolves “latest,” silently changes a historical result, or invents a normalized score when this exact scorer version has no reviewer-approved calibration binding with a canonical mappingVersionId.

​
Permissions and entry points

Open Measurement → Scorers (/measurement?view=scorers; the old /scorers link redirects there). Select a scorer to open its Scorer Studio workbench at /scorers/{scorerId}. All reads, writes, composite references, calibration references, results, and surface bindings are organization-scoped. A scorer or referenced version from another organization is rejected rather than disclosed.

​
Scorer types

Every type returns the same output contract: a finite score, a boolean passed, human-readable reasoning, and a structured evidence object. Deterministic regex definitions reject known nested-repetition patterns. Composite versions reject missing components, self-reference, organization mismatch, and content hashes that do not match the referenced immutable version. LLM judge definitions do not accept custom HTTP endpoints. Provider and model egress goes through the authoritative Model Gateway, which applies routing, provider policy, budgets, retries, failure classification, cost attribution, and the model-call ledger.

​
Author a definition

The Definition tab exposes the complete content-hashed JSON definition. It includes:
  • Input and output JSON Schemas
  • Implementation type and its full configuration
  • Numeric score minimum, maximum, pass threshold, and optional labels
  • Calibration publication policy and accountable owner
  • Tags and a concise purpose statement
  • Exact component or Prompt Hub version bindings where applicable
  1. Choose New scorer, enter a permanent lowercase key, and choose a deterministic, bounded TypeScript, or LLM judge template.
  2. Open the scorer and edit the complete definition.
  3. Resolve every contract issue shown next to the editor.
  4. Choose Create new version.
Saving does not mutate the selected version. It creates a lineage-linked draft with a new numeric ID and content hash. An identical canonical definition resolves idempotently to the existing content version rather than creating an indistinguishable duplicate.

​
Code execution boundary

Code packages declare allowNetwork: false and explicit limits:
The local development runner rejects module loading, host globals, dynamic code, filesystem access, process creation, and network APIs. It is restricted to non-production trusted development use and requires the explicit SCORER_TRUSTED_LOCAL_CODE_EXECUTION=1 opt-in. Production ignores that setting and never routes to the local executor. Production always routes code scorers through the OCI isolation boundary. Configure SCORER_OCI_RUNTIME as docker or podman and preload the digest-pinned Node and Python images declared by the release. The runner refuses floating image tags, verifies the checked-in seccomp profile hash, and has no production switch back to the local executor. The OCI process runs as UID/GID 65534, with a read-only root filesystem, no network or IPC namespace, no bind mounts, all Linux capabilities dropped, no-new-privileges, a no-exec temporary filesystem, and enforced process, CPU, wall-time, memory/swap, and output ceilings. A default-deny seccomp profile permits runtime threads but denies child-process creation. Only the scorer input, source, limits, and already-verified content/package identity cross stdin; host environment secrets are not copied into the container. Every run records the pinned image digest, seccomp profile hash, content hash, and package hash in its bounded system log. Missing runtimes or images, profile-integrity failure, timeout, output overflow, OOM, invalid output, and scorer exceptions remain distinct failure codes.

​
Build human anchors

The Examples tab stores append-only human anchors. Each anchor contains:
  • A name and optional review notes
  • The full scorer input: input, candidate output, optional expected output, context, latency, and cost
  • The expected human score on the selected scorer’s scale
  • Actor and creation time
Editing or deleting a persisted anchor is rejected. Create a new anchor when human judgment changes so historical alignment runs retain their exact input snapshot.

​
Test one exact version

The Test tab runs one advisory input against the selected versionId. Choose Playground, Experiment, or Online Eval and enter its resource ID. One-case tests cannot claim the Gate surface or establish release authority. Every stored result includes:
  • Organization, scorer artifact, exact version ID, and content hash
  • Surface and resource ID
  • Raw score, pass decision, reasoning, structured evidence, and captured logs
  • Explicit failure code/message, latency, and known or unknown cost
  • Model-call ID for an LLM judge
  • Canonical mapping-version ID plus its derived calibration set, run, and legacy mapping IDs when present
  • Comparability state and normalized score, or an explicit withheld value
  • Idempotency key and request hash
An LLM judge provider error, budget denial, timeout, or JSON parse failure remains a failed or blocked result with its model-call evidence. It is not presented as a successful score. Canonical evaluator validation executes Scorer versions through the same execution owner and records an exact validation slot, completed execution claim, and dependency receipt. Scorer validation supports pointwise protocols with one repeat. Pairwise and repeated Scorer protocols are rejected during planning. Composite receipts retain each executed child edge, including repeated references and optional-child failures. Consequential use checks the current qualification for the complete effective evaluator identity. An advisory test or publication approval does not establish gate authority.

​
Measure human alignment

The Human alignment tab executes a frozen anchor snapshot and returns an evidence matrix with: Completed alignment evidence is immutable. Repeating an idempotency key returns the original run instead of creating a conflicting final state.

​
Calibrate without hiding discontinuities

The Calibration tab separates a raw scorer output from an approved comparison scale. Comparability is one of:
  • exact_comparable: the exact canonical mapping version on the intended reference scale and an approved binding for this scorer version
  • calibrated_comparable: a reviewer-approved mapping version on a compatible anchor identity and score scale
  • trend_discontinuity: the change breaks an earlier score trend
  • not_comparable: no valid cross-version comparison is claimed
A comparable binding requires an exact canonical mappingVersionId linked to its approved anchor version, judge configuration, calibration run, legacy score mapping, and organization. EvalGate derives and validates the legacy IDs rather than trusting caller-supplied lineage. Supported mappings execute from their immutable stored breakpoints, percentile distribution, or z-score parameters.
Scorer Studio persists normalizedScore = null unless the exact version has an approved exact_comparable or calibrated_comparable mapping. Missing calibration is never treated as zero, one, or “close enough.”
An LLM judge publication policy can require human anchors, agreement and parse-failure limits, an approved calibration mapping, and an accountable owner. Required calibration also checks canonical class support and false-pass/false-fail risk; high aggregate agreement alone does not qualify an always-pass judge. Optional calibration permits advisory experimentation. Publication itself grants no consequential authority. Comparable scores require the live mapping to remain approved and free of disqualifying drift. Invalidating a dependency preserves historical results and their original identities.

​
Compare, review, and approve

The Versions tab lists every immutable version with its status, numeric ID, and content hash. Select a comparison version to inspect a line-level definition diff, including schemas, rubrics, model routing, scale, thresholds, code limits, component hashes, and calibration policy. The lifecycle is:
Submitting and approving are compare-and-set writes. The request includes the reviewRevision currently loaded in the workbench. A concurrent transition returns 409 STALE_VERSION; it does not overwrite the newer state. Every transition appends an actor, rationale, revision, timestamp, and idempotency key. Approval makes the exact version eligible for advisory publication or reuse. Consequential gate use additionally requires canonical evaluator qualification and action authorization.

​
Publish and bind consumers

Publishing requires an approved version, the loaded environment version/revision, and an idempotency key. The confirmation dialog names the current and target version IDs and content hash. The environment move and append-only event are atomic; concurrent stale promotions fail without a false event. The Usage tab binds an approved exact version to: Each binding persists surface, resourceId, versionId, contentHash, and the exact canonical mapping version available at creation. Usage results stay attached to the scorer version, mapping version, derived calibration lineage, and model call that produced them after a newer version ships. A Gate binding must match the scorer version on the run’s active canonical measurement release and pass its configured gate and action-authorization checks. The binding is an association; an enforcement receipt proves the protected boundary. Qualification requires the configured independent review, sealed holdout, approval scope, expiry, and critical-slice support and error ceilings. A good aggregate cannot compensate for an unmeasured or failing mandatory critical slice. Consequential use rechecks the live model, routing, rubric, parser, mapping, and effective composite dependencies. A changed or unavailable dependency requires renewed qualification. Legacy releases with no live source remain advisory (LIVE_EVALUATOR_SOURCE_REQUIRED); dependency drift returns EVALUATOR_DEPENDENCY_CHANGED or EVALUATOR_DEPENDENCY_UNAVAILABLE.

​
API workflow

Version creation, execution, alignment, review, approval, and publishing accept an Idempotency-Key header. Reusing a key for different intent returns a conflict.

​
Failure behavior

​
Troubleshooting

  • If version creation fails, fix the first nearby contract issue and validate the complete definition JSON.
  • If a composite fails, verify every component artifact/version/hash triplet and required flag.
  • If an LLM judge is blocked, inspect the model-call ledger for routing, budget, provider, or response-schema evidence.
  • If publish is blocked, compare the selected version’s human-alignment metrics with its calibration policy and verify that this scorer version’s calibration binding is approved.
  • If a normalized score is withheld, approve a canonical mapping version and bind its exact mappingVersionId to this scorer version; do not copy legacy calibration IDs or a binding from another version.
  • If production code execution is blocked, verify SCORER_OCI_RUNTIME, preload the exact Node/Python image digest, keep security/scorer-seccomp.json unchanged, and inspect the stored sandbox error and system log.
  • If a publication is stale, reload the workbench and review the newer environment revision before retrying.
Verification is maintained in the Scorer Studio unit, hostile-code OCI, API, database, DOM, and authenticated golden-path suites: tests/lib/scorer-studio.test.ts, tests/security/scorer-oci-sandbox.test.ts, tests/api/scorers, tests/integration/scorers, tests/dom/scorers, and e2e/scorer-studio-golden-path.spec.ts. The dedicated Scorer sandbox CI job pulls the same immutable images and runs the non-skipped adversarial boundary suite.