Skip to main content

​
Install an evaluation pack

Evaluation packs give you realistic, versioned coverage without starting from an empty dataset. EvalGate ships first-party packs for healthcare, legal, support, coding, and financial agents. Every pack contains multi-step scenarios, fictional source fixtures, criterion-level rubrics, adversarial variants, and critical-failure release rules. If you are starting from an existing codebase, first scan the repository, review the evidence-linked recommendations, and then choose a pack.

​
Install from the web app

  1. Open Evaluation packs at /eval-packs.
  2. Filter by domain and inspect the pack’s scenarios, evidence fixtures, criteria, and release-gate thresholds.
  3. Select Install pack.
  4. Open the created evaluation and run it against the candidate AI change.
Installation is atomic and retry-safe. EvalGate creates a published Dataset Hub version, immutable row snapshots, an active domain-pack evaluation, active-gate test cases, and a snapshot binding. It does not create a partial evaluation if installation fails.
First-party fixtures are fictional evaluation evidence. Packs help test a system; they are not legal, clinical, or financial advice and do not prove regulatory compliance by themselves.

​
Install from the CLI

The TypeScript package exposes the same workflow through client.evalPacks. The Python package exposes it through client.eval_packs.

​
Interpret a result

Domain packs score each criterion with its declared deterministic, trajectory, or semantic evaluator. Semantic criteria use the configured judge ensemble and must satisfy both the score and agreement thresholds. A candidate fails the release gate when a critical criterion fails, judge agreement is insufficient, or the required scenario pass rate is missed.

​
Next step

Run the created evaluation, inspect failing criteria and their cited fixtures, then use that evidence in the release decision.