> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Experiments

> Run immutable baseline-versus-candidate matrices, inspect paired evidence, approve an attributable winner, and create a governed publish proposal.

Experiments turn a mutable Playground comparison into a named, content-addressed evaluation record. Use them when a model, prompt, tool, scorer, routing policy, or dataset change needs reproducible evidence before release.

> [!NOTE]
>
> An Experiment never follows a mutable alias. It freezes the exact inputs available at creation time and executes those values again on every rerun.

## What an Experiment freezes

The snapshot is immutable and protected by database triggers. It includes:

* every evaluation test-case input, expected output, metadata value, lifecycle field, and source reference;
* the selected Dataset Hub version, schema, content hash, row order, row payloads, and row content hashes when a dataset version is attached;
* baseline and candidate prompt version identifiers, prompt content hashes, full messages, and system prompts;
* provider, model, routing profile and strategy, parameters, seed, and environment;
* tool configuration and policy snapshots;
* complete scorer definitions, scorer prompt hashes, and settings;
* calibration mapping identifiers, score scales, methods, and mapping payloads;
* evaluation execution settings, model settings, custom metrics, executor configuration, runner version, code version, and snapshot schema version.

The experiment record stores a SHA-256 hash of the canonical snapshot. Each variant has its own configuration hash. Changing a live prompt, Playground variant, dataset row, scorer, or calibration mapping after creation does not rewrite either value.

## Recommended workflow

**1. Prepare the matrix in Playground**

Create one baseline and at least one candidate. Select the exact provider, model, tools, parameters, scorers, and policy configuration that you intend to compare.

**2. Publish reusable inputs**

Publish the Dataset Hub version and approve any referenced Prompt Hub version. Draft or unapproved versions are rejected when the experiment is created.

**3. Promote to an Experiment**

Choose **Promote from Playground**, name the experiment, and record the expected dataset and variant hashes. If a live asset changes before promotion completes, creation fails with a `stale_asset` conflict instead of silently freezing a different value.

**4. Run the matrix**

Select **Run matrix**. Every variant receives a normal evaluation run, and every row receives a normal test result. Experiment bindings preserve the exact execution, variant, run, and test-result lineage.

**5. Inspect the comparison**

Review paired rows, regressions, score distributions, uncertainty, cost, p50/p95 latency, failures, tool errors, policy blocks, unknown values, and evidence completeness together.

**6. Record a decision**

Select a candidate and write a rationale. Winner approval is blocked when the candidate violates the frozen winner policy, introduces disallowed regressions, has incomplete cost, or crosses an uncalibrated judge boundary.

**7. Create a publish proposal**

An accepted winner can create a proposal for a prompt version, Evals-as-Code manifest, online monitor, release artifact, signed report, or release gate. The proposal is always `pending_approval`; the Experiment does not make a live change.

## Comparison semantics

Each candidate is paired to the baseline by frozen test-case identifier. The comparison reports:

| Dimension | Evidence shown |
| - | - |
| Quality | Baseline and candidate pass rates, delta, scorer distributions, score mean delta, and paired-row changes |
| Uncertainty | A 95% interval for the pass-rate difference when enough observations exist |
| Cost | Total known cost, delta, and separate unknown-cost row counts |
| Latency | Baseline and candidate p50/p95 plus both deltas |
| Failures | Failure-mode counts, new regressions, fixed regressions, tool-error delta, and policy-block delta |
| Provenance | Evaluation run, test result, model call, trace, evidence completeness, prompt/scorer hashes, and comparison hash |

Every candidate receives one outcome:

* `improvement`: pass rate or paired outcomes improve without a new paired regression;
* `regression`: quality falls or the candidate introduces a paired regression;
* `tradeoff`: the candidate fixes some rows and regresses others;
* `tie`: complete paired quality is unchanged;
* `inconclusive`: required paired data is missing or uncertainty crosses zero;
* `not_comparable`: judge or calibration identities differ without a shared scale.

The **Summary** tab shows the decision logic used for the classification. Missing data and judge discontinuities appear as warnings; neither is converted to zero.

## Experiment workspace

The detail workspace contains eight sections:

* **Summary** combines headline quality, cost, latency, classification, warnings, and decision logic.
* **Variants** shows the baseline and candidates with model, provider, scorer, calibration, and configuration hashes.
* **Rows** is a virtualized paired matrix with baseline output, candidate output, scorer delta, cost, latency, and classification.
* **Regressions** focuses on newly failing and newly fixed cases.
* **Metrics** shows distributions, uncertainty, operational tradeoffs, tool failures, and policy blocks.
* **Traces** links normal run, test-result, model-call, trace, and evidence identifiers.
* **Decision** records the candidate, comparison hash, rationale, actor, and decision time.
* **Evidence** shows publish proposals and the full frozen snapshot payload.

Row filters include regression, improvement, disagreement, unchanged, missing, high cost, tool error, policy block, and free-text category or tag search. The matrix only renders visible rows, so large datasets remain responsive without changing the comparison result.

## Progress, cancellation, and reruns

Execution progress reports completed variants and processed rows. A cancellation request stops the matrix before the next variant. Completed variant bindings remain append-only evidence.

Use **Rerun snapshot** to execute the same immutable values again. The new execution records its parent execution and increments the attempt number. Only one queued, running, or cancellation-pending execution can exist for an Experiment at a time.

> [!WARNING]
>
> Rerunning an Experiment does not refresh prompt aliases, dataset versions, scorer settings, calibration mappings, or policies. Create a new Experiment to test newer inputs.

## Empty and blocked states

| State | Meaning | Next action |
| - | - | - |
| `not_run` | No matrix execution exists | Run the frozen matrix |
| `partial` | A variant, paired row, cost, latency, or evidence link is missing | Inspect failed runs and rerun the same snapshot |
| `incomparable` | Judge or calibration identities do not share a scale | Create a calibrated mapping or a new comparable Experiment |
| `budget_blocked` | Estimated matrix cost exceeds the frozen budget policy | Reduce the matrix or create a new snapshot with an authorized budget |
| `stale_asset` | An expected hash changed before snapshot creation or run submission | Refresh the asset and explicitly create a new snapshot |
| No winner | No eligible accepted candidate exists | Record `needs_review` or `reject`; do not publish |

## API reference

All endpoints are organization-scoped. UUIDs from another organization or another Experiment are rejected by composite database constraints, even when the individual identifier exists.

| Method | Endpoint | Purpose | Required scope |
| - | - | - | - |
| `GET` | `/api/experiments` | List Experiments with optional evaluation and status filters | `eval:read` |
| `POST` | `/api/experiments` | Create an immutable API-configured snapshot | `eval:write` |
| `GET` | `/api/experiments/:id` | Read snapshot, variants, executions, decisions, and proposals | `eval:read` |
| `POST` | `/api/experiments/:id/run` | Execute or rerun the frozen matrix | `runs:write` |
| `POST` | `/api/experiments/:id/cancel` | Request cancellation of the active execution | `runs:write` |
| `GET` | `/api/experiments/:id/progress` | Read variant and row progress | `runs:read` |
| `GET` | `/api/experiments/:id/results` | Read normal run/test-result evidence with provenance | `runs:read` |
| `GET` | `/api/experiments/:id/compare` | Read paired comparisons and workspace state | `runs:read` |
| `POST` | `/api/experiments/:id/decision` | Record accept, reject, or needs-review evidence | `eval:write` and admin role |
| `POST` | `/api/experiments/:id/publish-winner` | Create an exact-snapshot publish proposal | `prompt:publish` and admin role |
| `POST` | `/api/evaluations/:id/playground/promote-to-experiment` | Freeze the current Playground matrix | `eval:write` |

### Idempotency

Creation, run, decision, and publish requests require an idempotency key. Replaying the same key and same request returns the existing record. Reusing a key with a different snapshot, rerun parent, winner, or target returns `409 CONFLICT`.

### Create an Experiment

```json theme={null}
{
  "name": "Checkout prompt v8",
  "description": "Candidate compared against the production prompt",
  "evaluationId": 42,
  "datasetVersionId": "c55e41d2-e234-45aa-8dd9-2ea901ad85ff",
  "expectedDatasetContentHash": "aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa",
  "baselineVariantKey": "production",
  "variants": [
    {
      "key": "production",
      "name": "Production",
      "promptVersionId": "101",
      "expectedPromptContentHash": "bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb",
      "provider": "openai",
      "model": "gpt-4.1-mini",
      "scorerConfigIds": [12],
      "calibrationMappingIds": ["mapping-2026-07"]
    },
    {
      "key": "candidate",
      "name": "Concise checkout",
      "promptVersionId": "108",
      "expectedPromptContentHash": "cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
      "provider": "openai",
      "model": "gpt-4.1-mini",
      "parameters": { "temperature": 0 }
    }
  ],
  "environment": "staging",
  "seed": 17,
  "runnerVersion": "evalgate-runner-v1",
  "codeVersion": "8c17162",
  "policies": { "budget": { "maxCostUsd": 25 } },
  "winnerPolicy": {
    "minimumPassRateDelta": 0,
    "maximumCostIncreasePercent": 15,
    "maximumP95LatencyIncreasePercent": 10,
    "allowIncomparableJudges": false,
    "allowRegressions": false
  },
  "idempotencyKey": "checkout-prompt-v8-create"
}
```

### Run and rerun

```json theme={null}
{
  "idempotencyKey": "checkout-prompt-v8-attempt-2",
  "parentExecutionId": "8c8151eb-39dc-43ae-9b74-c7b21ee75221",
  "expectedSnapshotHash": "dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd",
  "estimatedCostUsd": 12.4
}
```

Omit `parentExecutionId` for the first execution. A rerun parent must be terminal and belong to the same organization and Experiment.

### Approve a winner

```json theme={null}
{
  "decision": "accept",
  "winningVariantKey": "candidate",
  "rationale": "Pass rate improved by 4.2 points with no paired regressions; cost rose 3.1% and p95 fell 8%.",
  "comparisonHash": "the exact comparison hash shown in the workspace",
  "idempotencyKey": "checkout-v8-decision"
}
```

An accepted decision requires an eligible non-baseline candidate. `reject` and `needs_review` decisions must not provide a winning variant.

### Propose publication

```json theme={null}
{
  "decisionId": "e80a6b88-af07-4e70-a3cf-20d93ea11a2a",
  "target": "release_gate",
  "environment": "staging",
  "idempotencyKey": "checkout-v8-release-gate-proposal"
}
```

The response contains `status: "pending_approval"`, `apply: null`, the Experiment snapshot hash, winner configuration hash, comparison hash, and decision identifier.

## Audit evidence

The audit log records snapshot creation, execution completion or cancellation, winner decisions, and publish proposals. Use the following identifiers when exporting evidence:

* Experiment ID and snapshot hash;
* Dataset version and content hash;
* variant ID and configuration hash;
* execution ID, attempt, and rerun parent;
* evaluation run and test-result IDs;
* model-call, trace, and evidence identifiers;
* decision ID, actor, rationale, and comparison hash;
* publication proposal ID, target, environment, and `pending_approval` status.

These links allow an auditor to move from the release proposal back to the exact row output and frozen input that justified it.

## Related guides

* [Compare evaluators](https://evalgate.com/docs/guides/compare-evaluators.md) — Evaluator measurement protocols are not application Experiments
* [How the evaluation system fits together](https://evalgate.com/docs/concepts/evaluation-layers.md) — Application behavior versus evaluator measurement and authorization
* [Evaluation cost and latency](https://evalgate.com/docs/guides/evaluation-cost-and-latency.md) — Accounting boundaries for measurement and review assistance


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.