Skip to main content

​
Calibrate a scorer

Calibration helps answer whether scores from different scorer/model versions are comparable. A score such as 0.8 should not be assumed to mean the same thing after the measurement instrument changes. Calibration now lives under Measurement → Calibration at /measurement?view=calibration. The old /calibration route redirects to that view.

​
Step 1 — Build an anchor set

Use stable, independently reviewed examples spanning the quality range you care about. Version the anchor evidence so a later change cannot silently rewrite the basis of comparison.

​
Step 2 — Measure the scorer on the anchors

Run the exact scorer version against the anchor set and preserve the raw scores, labels, scorer identity, and environment/revision information needed to reproduce the comparison.

​
Step 3 — Review the mapping

The mapping translates raw scorer output into the normalized/comparable scale used by the calibrated measurement. Review the mapping and its evidence before approval.

​
Step 4 — Check comparability and drift

When the scorer, model, rubric, or other measurement input changes, compare the new mapping with the approved basis. If comparability is not established, keep the result explicitly incomparable or re-baseline rather than pretending the old threshold still means the same thing.

​
Step 5 — Bind the approved mapping

Bind the exact approved calibration/mapping version to the scorer/evaluation workflow that uses it. Release evidence should retain that version identity.

​
Up next

Try a candidate safely in the Playground.