> ## Documentation Index > Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Cli # EvalGate CLI command reference > Complete reference for EvalGate CLI authentication, repository linking, gates, CI integration, trace labeling, failure analysis, judge orchestration, and auto loops. The EvalGate CLI is the fastest way to run regression gates, analyze failure patterns, and automate prompt improvement without leaving your terminal. Run TypeScript CLI commands with `npx @evalgate/sdk ` for zero-install usage, or add `@evalgate/sdk` to your project and run the same commands through your package manager. The local CLI gate is **Experimental**, matching [feature status](https://evalgate.com/docs/platform/feature-status.md). Start with the [five-minute local gate](https://evalgate.com/docs/quickstart-typescript.md). In the first hour, use `gate`; [CI modes](https://evalgate.com/docs/guides/cicd-integration.md) compares later `ci` and hosted `check`, their evidence, credentials, costs and blocking reasons. Removed commands return targeted, non-mutating migration guidance in both CLI routers; use the migration table below to select the current command. They never silently provision credentials or apply a scaffold. ## Local core and hosted boundary The MIT-licensed CLI gives cloud coding agents a credential-free local execution path. `init --local` performs project discovery and scaffolding with no account, browser, organization, GitHub remote, API key, or network request and creates no hosted identity. Spec execution, baseline updates, and regression gates then run in the checkout. Browser-approved `login` and durable repository `link` are a separate optional hosted path. Exact commit and remote-head details remain cloud activation snapshots, not durable link identity. Baselines and local evidence remain reviewable files that can run in any CI provider. The hosted control plane is additive. Connect it for shared history, managed traces, team review, access control, and organization policy. Local gates remain available when the hosted service is not configured. ### Start a cloud-agent sandbox locally ```bash theme={null} theme={null} npx @evalgate/sdk init --local --format json npx @evalgate/sdk init --local --apply --format json npx @evalgate/sdk baseline update npx @evalgate/sdk gate --format json ``` Here `local` describes where evaluation executes; the intended caller may be a remote coding agent. JSON and non-TTY init is preview-only without explicit `--apply`. ### Add hosted repository collaboration The normal human path uses browser-approved device authorization and does not print or ask you to copy an EvalGate credential: ```bash theme={null} theme={null} npx @evalgate/sdk login npx @evalgate/sdk link npx @evalgate/sdk status ``` `link` detects the GitHub `origin`, current symbolic branch, exact local `HEAD`, and repository root. It previews the server-owned intake contract pinned to that exact target and asks before applying in a TTY. Branch movement does not unlink the project, and EvalGate never silently substitutes the remote branch head. For agents and CI, mutation remains explicit: ```bash theme={null} theme={null} evalgate status --json evalgate link --preview --format json evalgate link --apply --format json ``` Repository configuration cannot supply a key or redirect a user session to another API origin. After installation, deterministic `understand`, `init`, `health`, `baseline`, and `gate` commands can run without a network round trip. `status` keeps durable linkage, checkout movement, cloud snapshot, and readiness separate. A checkout behind its tracked branch can remain linked, locally gate-ready, and cloud-targetable. Ahead or unpushed work needs the exact commit pushed before cloud evidence; diverged work needs an explicit appropriate branch. Dirty worktree state is reported independently because Git SHA alone cannot reproduce uncommitted changes. > [tab] TypeScript (npx) > > ```bash theme={null} theme={null} > npm install @evalgate/sdk > npx @evalgate/sdk > ``` > [tab] Python > > ```bash theme={null} theme={null} > pip install evalgate-sdk > evalgate > ``` > [!NOTE] > > The lifecycle on this page follows the currently published `@evalgate/sdk` > package on npm, including credential-free `init --local` when that release > advertises it. Discover commands with `npx @evalgate/sdk --help` and > `capabilities --format json`; do not assume a source-tree command is > published. > The Python package `evalgate-sdk==3.13.1` on PyPI ships the same release > line; run `pip show evalgate-sdk` to confirm what is installed. ## Navigate by outcome Both CLIs expose the same versioned capability map. Start here when you know the job you need to accomplish but not the internal command name: ```bash theme={null} theme={null} # Human-readable map evalgate capabilities # One workflow with web routes, native commands, operation IDs, artifacts, and next step evalgate capabilities playground # Stable machine discovery for coding agents evalgate capabilities --format json ``` The complete map of governed core workflows covers local product understanding, setup, evaluation, Playground, Prompt Hub, Dataset Hub, first-party evaluation packs, repository intelligence, experiments, scorer development, logs, continuous evaluation, Failure patterns, dashboards, Copilot, provider onboarding, red-team testing, remote runners, deployable assets, governance, and release. TypeScript and Python emit identical JSON from the generated contract. Every mapped workflow names its web route, public operation IDs, native automation command, artifacts, and next governed step; generation fails when any required workflow disappears. ## Stable exit codes The process exit status is the shell/CI result. JSON reports may additionally carry `effectiveExitCode` and `releaseReady` when preview mode or evidence policy separates process completion from the release decision. The complete, version-bound contract is available from `evalgate capabilities --format json`. | Code | Symbol | Meaning | Release meaning | | - | - | - | - | | `0` | `PASS` | All required evidence passed the configured gate. | Release-ready when the report also says `releaseReady=true`. | | `1` | `SCORE_BELOW` | The measured score is below the configured minimum. | Blocks release. | | `2` | `REGRESSION` | Measured behavior regressed beyond the accepted baseline tolerance. | Blocks release. | | `3` | `POLICY_VIOLATION` | A configured policy check failed. | Blocks release. | | `4` | `API_ERROR` | The hosted API or network request failed. | Release readiness is inconclusive; retry or use an admissible cache. | | `5` | `BAD_ARGS` | The command arguments are invalid. | Blocks release until configuration is corrected. | | `6` | `LOW_N` | The measured sample is too small for the configured decision. | Blocks release unless policy explicitly allows weak evidence. | | `7` | `WEAK_EVIDENCE` | Evidence exists but is not strong enough for the gate. | Blocks release. | | `8` | `WARN_REGRESSION` | A metric entered the configured warning band. | Policy-dependent; never treat as a stronger pass than reported. | | `9` | `FAILURE_MODE_THRESHOLD` | A failure-mode threshold was exceeded. | Blocks release. | | `10` | `JUDGE_CREDIBILITY_UNTRUSTWORTHY` | Judge calibration is too weak to support a trusted decision. | Blocks release or remains inconclusive per policy. | | `11` | `IO_ERROR` | Required report or evidence could not be persisted. | Blocks release; a missing canonical report is not a pass. | | `12` | `BASELINE_TAMPER` | The baseline checksum or attestation is invalid. | Blocks release until the reviewed baseline is restored. | | `13` | `PROVIDER_UNAVAILABLE` | Required provider evidence was unavailable or stale. | Inconclusive or blocked; never release-green. | | `14` | `PROVIDER_INCONCLUSIVE` | Provider work was deferred under an explicit policy. | Inconclusive unless policy explicitly admits deferred evidence. | | `15` | `BASELINE_NOT_ACCEPTED` | The current run is measurable, but no reviewed baseline has been accepted. | Blocks release until a reviewed baseline is accepted. | | `99` | `INTERNAL_ERROR` | An unexpected internal error occurred. | Blocks release. | | `130` | `CANCELLED` | The operation was cancelled by the user or orchestrator. | No release decision was made. | ## Call any public API operation The TypeScript CLI's `api` command consumes the generated public operation registry. This gives automation a single command for every operation in the API reference without falling back to private routes or database helpers. Treat it as an escape hatch for advanced automation; `evalgate capabilities` and the native workflow commands are the primary navigation surface. ```bash theme={null} theme={null} # Inspect stable operation IDs and their methods, paths, and parameter contracts npx @evalgate/sdk api --list-operations --json # Read one page npx @evalgate/sdk api get_evaluations \ --query limit=25 \ --query offset=0 \ --json # Make an idempotent mutation npx @evalgate/sdk api post_evaluations \ --body '{"name":"Support quality"}' \ --idempotency-key create-support-quality-v1 \ --json # Traverse only pagination declared by the operation contract npx @evalgate/sdk api get_evaluations \ --paginate \ --limit 50 \ --max-pages 20 \ --json ``` Use `--path name=value`, `--query name=value`, and `--header name=value` more than once when an operation declares multiple inputs. `--body` accepts inline JSON or `@path/to/body.json`. Machine mode writes the exact successful API envelope to stdout; failures write a stable JSON error containing `code`, `message`, `status`, `requestId`, and `exitCode`. `Ctrl-C` aborts the active request and exits `130`. ### Red-team release evidence To supply a local release gate with durable Red-Team evidence, request the specific completed run that the release is based on: ```bash theme={null} theme={null} evalgate red-team evidence --run ``` The first-class command exists in both SDK CLIs and uses the generated public API operation. It validates the strict artifact and canonical hash, verifies the server's Ed25519 signature against `redTeam.expectedAttestationPublicKeySha256`, checks the configured organization, campaign snapshot, and exact release target, then atomically writes `.evalgate/red-team-evidence.json`. It compiles identity, completion, provider, attack, benign-utility, finding, remediation, golden, signed-report, and framework-reference evidence from that run. It does not execute a campaign or choose the latest run. A required local `redTeam` policy must pin the exact `artifactId`, `versionId`, and 64-character `contentHash` being released; the gate rejects missing, malformed, stale, mismatched, cross-organization, incomplete, or unverifiable evidence. Do not hand-author the JSON file or substitute a source-checkout digest for the hosted release target. A bare `redTeam.required: true` uses `pre_release_strict`: complete and fresh campaign evidence, attributable costs, compiler attestation, zero successful attacks, and 100% benign utility. It intentionally does not require a finding-scoped remediation report, so a clean first campaign can pass. Use `post_remediation_strict` when admitting a repaired finding; it additionally requires the baseline/fixed rerun comparison, no introduced fingerprints, active golden coverage, and an exact-hash signed report. The legacy `strict` name remains an alias for `pre_release_strict`. Pin `expectedTarget`, `expectedOrganizationId`, `expectedSnapshotHash`, `expectedAttestationKeyId`, and `expectedAttestationPublicKeySha256` in `evalgate.config.json`. Use `policyProfile: "custom"` only for a reviewed policy exception. A fresh CI job runs `evalgate red-team evidence --run ` before `evalgate gate`; it never receives the server's private signing key. In machine mode, export failures emit one JSON error envelope. Invalid arguments/configuration exit 5, API/auth/network failures exit 4, artifact integrity or identity-policy failures exit 3, local writes exit 11, and unexpected defects exit 99. Campaign creation, approval, and execution remain explicit actions requiring the appropriate Red-Team permissions. Fetching release evidence is a read-only compilation step. ## Setup and initialization > [accordion] evalgate understand - infer and confirm product context > > Reads repository evidence and immediately returns a working theory of the > product, its audience, the user's job, the primary visible output, likely > quality risks, confidence, unknowns, and one useful question. Inferred > risks remain hypotheses until a person confirms them or representative > behavior provides evidence. > > Quick mode is the default bounded scan of high-signal product files. Use > `--depth deep` or `--deep` for a broader local scan. Both modes preview > by default. > > > [tab] TypeScript (npx) > > > > ```bash theme={null} theme={null} > > # Immediate quick preview > > npx @evalgate/sdk understand > > > > # Broader machine-readable preview > > npx @evalgate/sdk understand --deep --format json > > > > # Save reviewed context with confirmed corrections > > npx @evalgate/sdk understand --apply \ > > --audience "Support agents" \ > > --job "Resolve policy questions accurately" > > ``` > > > [tab] Python > > > > ```bash theme={null} theme={null} > > # Immediate quick preview > > evalgate understand > > > > # Typed lifecycle events during a deeper scan > > evalgate understand --depth deep --format jsonl > > > > # Save reviewed context with confirmed corrections > > evalgate understand --apply \ > > --audience "Support agents" \ > > --job "Resolve policy questions accurately" > > ``` > > Human output is the default. `--format json` returns the complete plan > envelope, while `--format jsonl` emits typed scan, evidence, theory, risk, > question, and completion events for agents and automation. `--json` is an > alias for JSON output. > > Preview writes nothing. It performs no network or model calls, executes no > project commands, inspects or changes no Git state, and does not modify > baselines, promote cases, or install telemetry. After review, `--apply` > writes the canonical context to `evalgate.product.json`. A custom > `--output` must remain inside the selected project root. > > > [!NOTE] > > > > Product understanding is available in the current source tree. > > TypeScript and Python CLI parity is locally verified, but package-release > > evidence, REST and mounted web authoring, marketplace distribution, and > > controlled coding-agent compatibility proof are still pending. > [accordion] npx @evalgate/sdk init — preview and scaffold a project > > Detects Node, Python, polyglot, and nested workspace projects. `init --local` is the credential-free cloud-agent path and creates no hosted context; plain `init` is the hosted-linked variant. In a TTY bare init previews and asks before applying. Explicit `--preview` never asks or writes. JSON and non-TTY calls remain non-mutating unless `--apply` is explicit. Init creates a pending baseline without executing tests or accepting results. It never mints or accepts an implicit activation key. > > The JSON plan includes `project.aiUsage` with evidence-backed agent frameworks, model providers, observability libraries, and tool protocols. Declared dependencies are high confidence; exact source imports are medium confidence. `init --apply` preserves the same evidence under `agentRuntimeRegistry.discovery` in `evalgate.config.json` and chooses a default runtime only when one framework is unambiguous. > > Discovery is local, bounded, and deterministic. It ignores dependency/build/virtual-environment directories, does not read `.env` files, and never guesses credentials or model names. > > ```bash theme={null} theme={null} > npx @evalgate/sdk init --local # credential-free interactive preview > npx @evalgate/sdk init --local --format json # cloud-agent preview > npx @evalgate/sdk init --local --apply --format json # explicit cloud-agent write > npx @evalgate/sdk init # hosted-linked interactive variant > npx @evalgate/sdk init --preview # explicit non-mutating human preview > npx @evalgate/sdk init --preview --format json # automation preview > npx @evalgate/sdk init --apply # explicit automation write > npx @evalgate/sdk baseline update > ``` > > Supported first-class handlers are npm, pnpm, the legacy `yarn` alias, > Yarn Classic, Yarn Modern, Bun, Deno, pip, uv, Poetry, Pipenv, PDM, > Conda, Mamba, Hatch, Pixi, and explicit custom JSON argv. Conflicting > JavaScript or Python/environment markers stop the plan. Repeat > `--package-handler ` for an explicit polyglot composition; gate and > baseline update execute every runnable handler from its owning root. > > ```bash theme={null} theme={null} > npx @evalgate/sdk init --package-handler bun --package-handler uv > ``` > > `baseline update` accepts only successful current evidence from the exact > clean source that was evaluated. A pending baseline is not release-ready, > and generated CI never updates, commits, or pushes one. The local gate > writes its canonical release decision to `.evalgate/last-report.json` and > detailed deltas to `evals/regression-report.json`. > > Then commit the generated files and push to trigger your first CI gate: > > ```bash theme={null} theme={null} > git add evals/ .github/workflows/evalgate-gate.yml evalgate.config.json > git push > ``` > [accordion] Migrate removed 3.8.x commands > > Removed command families fail without mutation using `BAD_ARGS` (exit 5) > and targeted next actions. JSON mode emits one document and redacts > secret-like arguments. > > | Removed form | Current path | > | - | - | > | `auth status` | `evalgate whoami` for identity; `evalgate status` for readiness | > | `auth configure --api-key ...` | Human: `evalgate login`; hosted automation: `EVALGATE_API_KEY`; local execution: `evalgate init --local` | > | `setup status` | `evalgate status` | > | `setup --repository owner/name` | `evalgate link --repository owner/name` after login | > | `start` | `evalgate watch` for continuous execution, or the explicit init/gate command for the intended outcome | > | `verify` | `evalgate doctor --quick` for diagnostics; `evalgate gate` for the release decision | > > An API key authorizes automation against an already established hosted > context. It cannot bootstrap hosted repository linkage and is unnecessary > for the local gate. > [accordion] npx @evalgate/sdk doctor — environment diagnostics > > Diagnoses local project readiness without cloud access in quick mode. Omit > `--quick` when you also want account and platform connectivity checks. > > ```bash theme={null} theme={null} > npx @evalgate/sdk doctor --quick > ``` > [accordion] npx @evalgate/sdk health — on-demand application health > > EvalGate is not limited to AI applications. A small local manifest can check > safe HTTP routes, webhooks, background jobs, or command-based smoke tests. > Checks run sequentially only when a developer or CI job calls them; no > collector, daemon, cloud credential, or long-lived connection is required. > > ```bash theme={null} theme={null} > npx @evalgate/sdk health --config evalgate.application.json --format github > ``` > > The JSON report contains bounded latency metrics, status/error counts, > redacted reasons, and a suggested inspection path. In GitHub Actions, > `--format github` appends the measured reasons to the PR Step Summary. The > repository `evalgate-application-health` Skill helps a coding agent create > the manifest and prepare a reviewed change; it does not edit or merge code > automatically. > > Python projects use the parity command: > > ```bash theme={null} theme={null} > evalgate health --config evalgate.application.json --format github > ``` > [accordion] npx @evalgate/sdk harness — optimize agent work locally > > To evaluate delegation with the versioned evidence extension, see > [Evaluate your subagents](https://evalgate.com/docs/guides/evaluate-your-subagents.md). The existing > command accepts an independent expected-root frame: > > ```bash theme={null} > evalgate harness optimize --input .evalgate/harness-events.jsonl \ > --frame frame.json --output report.json --format json > ``` > > These comparisons are local and exploratory. Child sessions and retries > are not independent root outcomes. Use a build containing the extension; > generic exports do not certify native provider context inheritance. > > EvalGate can improve the harness used to build or operate an application, > not only the application's own AI behavior. Give the CLI a bounded JSONL > file containing session outcomes and counters such as input/output tokens, > repeated context, redundant tool calls, Skill loads, and cache hits. > > Each JSONL line contains counters only, for example: > > ```json theme={null} theme={null} > {"schemaVersion":1,"sessionId":"session-001","harnessKey":"claude-code","agent":"claude-code","outcome":"passed","inputTokens":1200,"outputTokens":240,"repeatedContextTokens":300,"discoveryTokens":180,"toolCalls":8,"redundantToolCalls":2,"explanationTokens":40,"skillLoads":4,"usefulSkillLoads":3,"cachedInputTokens":200,"cacheableInputTokens":600,"costUsd":0.04,"recordedAt":"2026-08-15T12:00:00Z"} > ``` > > ```bash theme={null} theme={null} > npx @evalgate/sdk harness optimize \ > --input .evalgate/harness-events.jsonl \ > --output .evalgate/harness-report.json \ > --format github > ``` > > The local report identifies likely waste such as repeated repository > discovery, eager Skill loading, redundant reads, or overly long passing > summaries. It waits for five outcome-labeled sessions, refuses to optimize > when quality is below 70%, and shows a conservative token estimate. It > never reads prompts, completions, tool arguments, traces, credentials, or > starts a collector. It proposes a change; the existing eval suite must > validate the base and candidate before the change is considered validated. > > ```bash theme={null} theme={null} > npx @evalgate/sdk harness feedback \ > --report .evalgate/harness-report.json \ > --recommendation \ > --decision accepted|rejected|validated > ``` > > Python exposes the same commands as `evalgate harness optimize` and > `evalgate harness feedback`. `evalgate init --apply` installs the portable > `evalgate-harness-optimization` Skill in `.agents/skills/` so a coding agent > can explain the result and prepare the change without requiring a cloud > connector. > > After the base-vs-candidate eval has passed, record `validated` feedback, > then plan one managed Skill edit: > > ```bash theme={null} theme={null} > npx @evalgate/sdk harness mutate \ > --report .evalgate/harness-report.json \ > --recommendation \ > --skill .agents/skills/evalgate-harness-optimization/SKILL.md \ > --history .evalgate/harness-feedback.jsonl \ > --format json > ``` > > Apply requires the explicit `--apply` authority flag and the matching > validated report hash. EvalGate changes only its own deterministic managed > block, refuses stale/out-of-repository/symlink targets, and writes rollback > metadata under `.evalgate/harness-mutations/`: > > ```bash theme={null} theme={null} > npx @evalgate/sdk harness mutate \ > --report .evalgate/harness-report.json \ > --recommendation \ > --skill .agents/skills/evalgate-harness-optimization/SKILL.md \ > --history .evalgate/harness-feedback.jsonl \ > --apply \ > --format json > > npx @evalgate/sdk harness rollback \ > --mutation .evalgate/harness-mutations/.json > ``` > > Python exposes the same mutation and rollback commands. Mutation remains > local and on-demand: no daemon, cloud credentials, raw source snapshot, > prompt, completion, or unrelated instruction-file rewrite is involved. ## Gate and CI ### Stability policy for probabilistic checks An LLM call or model-judge verdict is not deterministic just because it runs in CI. Build hard gates on deterministic assertions where possible. For probabilistic checks, pin model and judge configuration, execute independent trials with the SDK's `repetitions` option, set an explicit `minimumPassRate`, and fail or quarantine evidence reported in `flakyCases`. Retries are transient recovery attempts within a trial; they are surfaced as flaky evidence and should not be used to retry a verdict until it passes. A seed makes EvalGate's case ordering reproducible but cannot prevent an external provider from changing a model. Recalibrate release-blocking judges against labeled examples, require the CLI's judge TPR/TNR and minimum-sample thresholds, and keep the resulting credibility evidence with the gate report. Exit code `10` means the configured judge is not credible enough to block a release safely. Before any provider call, the auto editor and SDK model-assertion boundary reject common instruction overrides, role spoofing, and judge-score coercion in untrusted evaluation text. A security rejection fails closed and never falls back to an automatic prompt mutation. The `check --onFail import` network boundary recursively redacts nested secrets and common PII from the complete failed-run envelope; this convenience upload path has no raw-data opt-out. > [accordion] npx @evalgate/sdk gate — run the regression gate locally > > Compares your current test results against the stored baseline and exits non-zero when the gate fails (for example `2` for a regression; see [Stable exit codes](#stable-exit-codes)). > > ```bash theme={null} theme={null} > npx @evalgate/sdk gate > ``` > > Output format options: > > ```bash theme={null} theme={null} > npx @evalgate/sdk gate --format github # CI step summary and job annotations > npx @evalgate/sdk gate --format json # Machine-readable JSON output > ``` > [accordion] npx @evalgate/sdk ci — one-command CI gate > > Discovers eval specs, runs them, writes results, and compares against a base run when `--base` is provided. Add `--impacted-only` to run only specs affected by the current diff. > > ```bash theme={null} theme={null} > npx @evalgate/sdk ci --format github --write-results --base main > ``` > > Full GitHub Actions workflow: > > ```yaml theme={null} theme={null} > name: EvalGate CI > on: [push, pull_request] > jobs: > evalgate: > runs-on: ubuntu-latest > steps: > - uses: actions/checkout@v4 > - uses: actions/setup-node@v4 > - run: npm ci > - run: npx @evalgate/sdk ci --format github --write-results --base main > - uses: actions/upload-artifact@v4 > if: always() > with: > name: evalgate-results > path: .evalgate/ > ``` > [accordion] npx @evalgate/sdk check — platform gate > > Runs the regression gate against the EvalGate platform (requires `EVALGATE_API_KEY`). Use `--onFail import` to upload failed run context to the dashboard for review. > > ```bash theme={null} theme={null} > npx @evalgate/sdk check --format github --onFail import > ``` > [accordion] npx @evalgate/sdk baseline update — refresh the baseline > > Re-runs your tests and accepts the results as the new baseline only when they pass from a clean checkout. Run this after you intentionally change model behavior or fix a known issue. > > ```bash theme={null} theme={null} > npx @evalgate/sdk baseline update > ``` ## Labeling and analysis > [accordion] npx @evalgate/sdk label — interactive trace labeling > > Steps through your unlabeled traces one by one. Use arrow keys to select pass/fail and pick a failure mode. Press `u` to undo the previous label. Press `Ctrl-C` to save progress and exit. > > ```bash theme={null} theme={null} > npx @evalgate/sdk label > ``` > > Each label you save becomes part of the golden dataset that can be used by later eval runs and gates. > [accordion] npx @evalgate/sdk analyze — failure-mode frequency report > > Aggregates labeled traces and prints a frequency report of failure modes (counts and share of labeled failures). > > ```bash theme={null} theme={null} > npx @evalgate/sdk analyze > npx @evalgate/sdk analyze --format json > ``` > [accordion] npx @evalgate/sdk failure-modes — configure failure-mode taxonomy > > Lists standard failure modes used by `label`, and optionally seeds or extends your `evalgate.config.json` alert weights. > > ```bash theme={null} theme={null} > npx @evalgate/sdk failure-modes > npx @evalgate/sdk failure-modes --init > npx @evalgate/sdk failure-modes --add my_custom_mode > npx @evalgate/sdk failure-modes --json > ``` > > Custom modes are stored under `failureModeAlerts.modes` in config and used by `analyze` and gate alerting. > [accordion] npx @evalgate/sdk replay-candidate — inspect a stored candidate for replay > > Loads a candidate eval case from the platform and prints the minimized input for manual or scripted re-run. The legacy `replay` alias is deprecated — prefer `replay-candidate`. > > ```bash theme={null} theme={null} > npx @evalgate/sdk replay-candidate > npx @evalgate/sdk replay-candidate 42 --format json --model gpt-5.2-chat-latest > ``` > [accordion] npx @evalgate/sdk replay-decision — compare two runs > > Loads two saved run artifacts and emits a keep/discard decision for each case — useful for reviewing whether a prompt change improved or regressed specific failure modes. > > ```bash theme={null} theme={null} > npx @evalgate/sdk replay-decision \ > --previous .evalgate/runs/run-prev.json \ > --current .evalgate/runs/run-latest.json > ``` ## Agent evidence and enterprise controls > [accordion] npx @evalgate/sdk run-cli — run a command with evidence capture > > Wraps an agent or CLI subprocess, captures stdout/stderr and tool events, and posts an evidence envelope to EvalGate (unless `--local`). > > ```bash theme={null} theme={null} > npx @evalgate/sdk run-cli \ > --runtime support-agent \ > --permission-profile default \ > -- npm test -- --grep "billing" > ``` > > Useful flags: > > ```bash theme={null} theme={null} > npx @evalgate/sdk run-cli --runtime my-runtime -- -- my-agent.sh > npx @evalgate/sdk run-cli --local --format json -- my-agent.sh > npx @evalgate/sdk run-cli --strict-persistence --evidence-status .evalgate/run-cli-status.json -- my-agent.sh > ``` > [accordion] npx @evalgate/sdk mcp — inspect, audit, or proxy MCP servers > > Validates MCP manifests, audits connector readiness, or proxies a server while emitting evidence. > > ```bash theme={null} theme={null} > npx @evalgate/sdk mcp inspect --manifest ./mcp/manifest.json > npx @evalgate/sdk mcp audit --connector filesystem --manifest ./mcp/manifest.json > npx @evalgate/sdk mcp proxy --runtime mcp-host --server filesystem -- npx @modelcontextprotocol/server-filesystem /tmp > ``` > > Add `--format json` for machine-readable audit output. Critical findings block by default; use `--fail-on high` (or `low`, `medium`, `critical`) to set the CI threshold. Use `--strict-persistence` separately when evidence must land on the platform. > [accordion] npx @evalgate/sdk controls — inspect enterprise guardrails > > Fetches org-scoped system controls: provider allowlists, PII scrub mode, retention policy, drift alerts, and guardrail state. > > ```bash theme={null} theme={null} > npx @evalgate/sdk controls > npx @evalgate/sdk controls --format json > ``` > > Requires `EVALGATE_API_KEY` with access to `/api/system/controls`. > [accordion] npx @evalgate/sdk share — create a share link for a run > > Creates a time-limited share link for an evaluation run. Use `--format json` to emit the token and URL for automation. > > ```bash theme={null} theme={null} > npx @evalgate/sdk share \ > --evaluationId 42 \ > --runId 9001 \ > --expires 7d \ > --format json > ``` ## Repository intelligence and evaluation packs Use the local activation path first. After the gate is working and GitHub is connected, scan an exact commit, review the evidence, and then install coverage: ```bash theme={null} theme={null} npx @evalgate/sdk repo repositories npx @evalgate/sdk repo scan --repository 42 --head-sha <40-character-sha> npx @evalgate/sdk repo ask --repository 42 \ --question "What AI models, agents, tools, and evals exist?" npx @evalgate/sdk packs list --domain coding npx @evalgate/sdk packs install coding-agent-release-safety ``` Repository scanning never executes repository code. Source findings and pack recommendations remain evidence-bounded and report incomplete scans honestly. ## Advanced > [accordion] npx @evalgate/sdk cluster — group similar failures > > Reads a saved run artifact and groups cases with similar failure patterns. Use the output to prioritize which failure mode to fix first. > > ```bash theme={null} theme={null} > npx @evalgate/sdk cluster --run .evalgate/runs/latest.json > ``` > [accordion] npx @evalgate/sdk synthesize — draft quarantined eval cases > > Start with contrastive response examples or expand a labeled failure > dataset. Profile mode previews locally without writing or calling a model. > > ```bash theme={null} theme={null} > # Examples-first preview, then explicit apply > npx @evalgate/sdk synthesize \ > --profile evalgate.quality.json \ > --format json > npx @evalgate/sdk synthesize \ > --profile evalgate.quality.json \ > --apply > > # Existing labeled-failure workflow > npx @evalgate/sdk synthesize \ > --dataset .evalgate/golden/labeled.jsonl \ > --output .evalgate/golden/synthetic.jsonl > ``` > [accordion] npx @evalgate/sdk auto — bounded autonomous prompt-improvement loop > > Reads labeled failures and prior prompt history, generates the next candidate prompt edit, evaluates it against the specs that depend on the prompt, and keeps the edit only if none of those compared specs regress. The loop terminates on explicit guard conditions rather than running open-ended. > > `auto run` and every `auto daemon` cycle execute with `.evalgate/auto/program.md` itself, so its mutation families, adaptive-loop, swarm and stop conditions apply to unattended cycles too. `budget.max_experiments` caps a run; `daemon.max_experiments_per_cycle` sets the per-cycle budget. > > The loop splits the cases into three populations by source family (a `family:` tag, otherwise the spec file), so cases derived from the same source never straddle populations. `holdout.ratio` sizes the **selection** population and `holdout.confirmation_ratio` the **confirmation** population; everything else is **development**. The assignment is written once to `.evalgate/auto/populations.json` and existing families never move; new families are placed as they appear. > > * The editor sees failures from development cases only. > * Baseline and candidates run development and selection cases. A candidate that regresses any selection case is discarded, whatever its development gain. > * Confirmation cases are never run, shown or read by the local loop. They are left for an independent confirmation of the frozen candidate. > > The ledger (`.evalgate/auto/ledger.jsonl`, schema 2) records whether each dimension was measured. The selection population is recorded as the measured holdout, with its regression count; when there are too few source families to reserve one, the holdout is recorded as unavailable rather than as zero regressions. Cost is recorded only when the runs report it (for example with a cost budget source); otherwise it is unavailable, and `auto history` reports that spend as not measured instead of counting it as zero. Rows written by schema 1 are read back with holdout and cost unavailable. > > When `program.md` contains `review: required: true` (the `auto init` template does), `auto run` refuses to start until the plan is approved. `auto review` prints the plan: populations (confirmation only as counts and a digest), graders, protected slices, mutation target and spend bound, each with a risk level. `auto review --approve` records who approved which plan hash in `.evalgate/auto/review.json`; high-risk items must be accepted explicitly with `--accept-risk `. Any change to the plan (a new case, a different budget) requires a new approval. > > ```bash theme={null} theme={null} > npx @evalgate/sdk auto review > npx @evalgate/sdk auto review --approve --accept-risk unbounded-spend > ``` > > Every run is held to an **Optimization Contract**. `auto contract --intent "…"` compiles what you want ("Fix tone mismatch in the prompt. Keep accuracy within 1%. At most 5 experiments. Spend up to \$3.") together with program.md and the current measurement into `.evalgate/auto/contract.json`; without stated intent the contract is compiled from program.md. The contract is part of the reviewed plan, and `auto run` enforces it while executing: > > * Only the editable prompt may change. Spec files, grader helpers (`dependsOn.code`), datasets, `evals/baseline.json`, `program.md` and `evalgate.config.json` are immutable. If they changed since the contract was compiled, the run refuses to start. If a candidate's execution changes any of them, the candidate is refused, the approved files are restored, and the refusal is kept in `.evalgate/auto/details/*-execution.json` (no ledger score is recorded). > * The run may not ask for more trials than the contract allows. Once execution spend exceeds a stated limit, or can no longer be measured against it, the loop stops before the next candidate. > * Confirmation cases are never requested. > * Authority is fixed: the loop proposes and tests, EvalGate qualifies, and a person adopts and merges. Clauses the compiler does not understand are listed in the review and must be accepted by name (`--accept-risk contract-unrecognized-intent`). > > Each ledger row records the `contractHash` it ran under. > > After a run, hand the kept candidate to people and to EvalGate: > > * `auto freeze` pins the kept candidate by content hash. It commits the candidate to an isolated branch (`evalgate/auto/`) through a temporary git worktree and restores your working tree to the base content. If the working-tree prompt is not byte-identical to a kept candidate, the freeze is refused. > * `auto report` writes `.evalgate/auto/report.md`. It covers every candidate tried and its hypothesis, and development and selection pass rates with 95% intervals. It also includes cost and latency with their boundaries, cheaper candidates that failed and the cases they broke, the exact sentences added and removed, spend, and limitations. The report states that the candidate is **not confirmed**. > * `auto export` writes `.evalgate/auto/campaign.json` (`evalgate.auto-campaign/v1`). It carries the base and candidate commits and content hashes, the contract, and the development and selection observations. Confirmation cases appear only as a count and a digest. > * `auto propose` runs all three. It prints a draft pull request command; `--push` pushes the branch and `--open-pr` opens the draft with `gh`. Nothing merges or deploys. > > Import the export into EvalGate to confirm the candidate independently. EvalGate binds the candidate commit, then runs approved cases of a remote-runner evaluation on the base and candidate target versions. Only when quality holds on cases the optimizer never saw does the journey show the candidate as confirmed, with the cost claim checked from runner execution evidence. A person then starts release review and merges the pull request. > > ```bash theme={null} theme={null} > npx @evalgate/sdk auto freeze > npx @evalgate/sdk auto propose --push > ``` > > **Scheduled campaigns.** `auto dispatch` is the command a scheduler calls (a nightly CI job, a cron). It runs the reviewed plan exactly as approved, in a detached git worktree of the committed base with a copy of `.evalgate/`, so your checkout is never touched. It refuses with exit code 3 instead of running when: > > * program.md does not require review, or no plan is approved; > * the plan changed since it was approved (cases, graders, budgets, contract or measured files), or the approval is older than `--approval-max-age-days` (default 14); > * tracked files have uncommitted changes; > * another dispatch holds the lock (a lock left by a dead or expired dispatch is recovered and that dispatch is recorded as abandoned); > * this approval was already dispatched `--max-per-approval` times (default 3). > > A scheduled checkout only sees what is committed, so commit the approved state: `.evalgate/auto/program.md`, `populations.json`, `contract.json` and `review.json`. Approving again (`auto review --approve`) and committing `review.json` is how a person renews or changes what the schedule may run. > > **Dispatch state must outlive the checkout.** Idempotency, the per-approval cap and overlap protection are only as durable as the record of earlier dispatches. A CI job starts from a fresh checkout, so `--state-remote origin` keeps that record as refs on the remote (`refs/evalgate/dispatch/*`): the lock, one record per dispatch, and cancel requests. Every write is a compare-and-swap push, so two scheduled jobs cannot both take the lock or the same attempt. Under CI (`CI` set) dispatch refuses with `EPHEMERAL_STATE` without `--state-remote`; pass `--local-state` only when the workspace persists between runs (a workstation, a self-hosted runner with a persistent workspace). If the remote cannot be read or written, dispatch refuses with `STATE_UNAVAILABLE` and nothing runs. > > The same approval and commit are dispatched once; a second trigger is a no-op. `--max-runtime-minutes` (default 120) stops a run on top of the contract's trial and spend limits, and `auto dispatch --cancel` (with the same `--state-remote`, from any machine) stops the running one. Stopping terminates the loop's whole process tree, including eval and provider subprocesses: SIGTERM, then SIGKILL after 10 seconds, and the worktree is removed only after every process has exited. A kept candidate is frozen on its own branch with a report and export under `.evalgate/auto/dispatches//`; `--open-pr` pushes it and opens a **draft** pull request. Nothing merges. `auto dispatch --status` lists recent dispatches. > > ```yaml theme={null} theme={null} > # .github/workflows/evalgate-campaign.yml > on: > schedule: > - cron: "17 3 * * 1-5" > concurrency: > group: evalgate-campaign > cancel-in-progress: false > jobs: > campaign: > runs-on: ubuntu-latest > permissions: > contents: write > pull-requests: write > steps: > - uses: actions/checkout@v4 > - run: npm ci > - run: npx @evalgate/sdk discover --manifest > # State lives on the remote: this checkout is new every night. > - run: npx @evalgate/sdk auto dispatch --open-pr --state-remote origin > env: > GH_TOKEN: ${{ github.token }} > ``` > > When the contract's objective is cost ("Cut cost by changing the prompt. Keep accuracy within 5%"), each candidate is judged on measured operating cost and preserved quality instead of a failure-mode delta. If cost is unknown on either run, the candidate is never kept as a saving. A candidate that is not cheaper, or that loses more pass rate than the protected margin, is discarded. A cheaper candidate within the margin is kept as selection evidence only. The cost claim itself needs per-case non-inferiority and independent confirmation, which EvalGate decides when the frozen candidate is qualified. > > ```bash theme={null} theme={null} > npx @evalgate/sdk auto contract --intent "Fix tone mismatch in the prompt. Keep accuracy within 1%. At most 5 experiments" > npx @evalgate/sdk auto review > ``` > > ```bash theme={null} theme={null} > npx @evalgate/sdk auto \ > --objective tone_mismatch \ > --prompt prompts/support.md \ > --autonomous \ > --budget 3 > ``` > > To repeat bounded cycles unattended (for example, overnight): > > ```bash theme={null} theme={null} > npx @evalgate/sdk auto daemon --cycles 5 > ``` > > > [!NOTE] > > > > The program-driven loop above is implemented in the TypeScript CLI (`npx @evalgate/sdk auto`). The Python CLI's `evalgate auto run` and `evalgate auto daemon` do not read `program.md`; they build a report from run artifacts you supply. > [accordion] npx @evalgate/sdk import inspect — check an external optimization result (conditional) > > External-import intake is **conditional on the installed CLI**. Discover > capabilities first: > > ```bash theme={null} theme={null} > npx @evalgate/sdk capabilities --format json > npx @evalgate/sdk --help > ``` > > `import` first shipped in `@evalgate/sdk@3.12.5`; 3.12.4 and earlier do > **not** include it. If `import` is absent from the installed help and > capability map, preserve the artifact and report the unsupported boundary; > do not invent a hosted import route. > > When `import` is advertised, `import inspect` reads a result another tool > produced and checks it independently. Supported today: Anthropic > `hillclimb/v2` (`state.json` from a build-eval or hillclimb run). EvalGate > recomputes every score from the per-case observations and reports what the > evidence supports and what it cannot establish. > > ```bash theme={null} theme={null} > npx @evalgate/sdk import inspect .claude/hillclimb/my-flow/state.json > npx @evalgate/sdk import inspect state.json --format json > ``` > > An import is something EvalGate received, not something it proved: its authority is investigation only, it is not EvalGate evidence, and it never authorizes a release. The output separates observation integrity from the imported claim: a reported winner that the observations contradict is shown as `DISAGREES WITH OBSERVATIONS`, and EvalGate uses the recomputed winner. The paired delta is reported per case (reps averaged within each case, attempts counted separately) with a descriptive 95% interval, because the source cannot establish case independence and the candidate was selected on those cases. A content-addressed diff identifies the imported change, not the program that ran. Reported experiment spend, recomputed spend, candidate operating cost and verification cost are reported separately and never combined. A `test` split is a label and not a sealed holdout, a report hash identifies files and not the program that produced them, errored attempts are kept out of scores, and cost without model and token usage is unverified. Duplicate `(case, rep)` rows or undeclared variants are refused and nothing is imported. The command is local and read-only: no network, no credentials, no files written. Exit `0` means an assessment was produced, not that a candidate is ready; `5` means the document was refused; `11` means the file could not be read. > [accordion] npx @evalgate/sdk discover --manifest — refresh the spec manifest > > Scans your project for eval spec files, refreshes the manifest, and reports any redundant or overlapping specs. Run it after creating, editing, renaming, or deleting any eval file: until the manifest matches the files on disk, `baseline update` and `gate` refuse with `COVERAGE_NOT_REGISTERED` (exit 7) and name the untracked or stale files. Refreshing the manifest does not accept a baseline; review and commit the eval source, then run `baseline update`. > > ```bash theme={null} theme={null} > npx @evalgate/sdk discover --manifest > ``` ## Judge commands > [accordion] npx @evalgate/sdk judge registry — list available judges > > Prints all judges available in the EvalGate registry for your organization. > > ```bash theme={null} theme={null} > npx @evalgate/sdk judge registry > ``` > [accordion] npx @evalgate/sdk judge presets — list judge presets > > Prints the built-in judge presets (pre-configured provider + model + prompt combinations). > > ```bash theme={null} theme={null} > npx @evalgate/sdk judge presets > ``` > [accordion] npx @evalgate/sdk judge test — test a judge configuration > > Runs a judge against a single input/output pair and prints the score, reasoning, and signals. Use this to validate a judge configuration before wiring it into your gate. > > ```bash theme={null} theme={null} > npx @evalgate/sdk judge test \ > --provider openai \ > --model gpt-5.2-chat-latest \ > --judge support_quality \ > --input "Cancel my subscription" \ > --output "I've canceled your plan effective today." > ``` > > Example output: > > ```json theme={null} theme={null} > { > "score": 0.92, > "passed": true, > "reasoning": "The response directly addresses the user's request with a clear confirmation.", > "signals": ["direct", "action_confirmed", "professional_tone"] > } > ``` > [accordion] npx @evalgate/sdk judge compare — compare two outputs > > Runs a judge against two candidate outputs for the same input and returns a preference decision with reasoning. Useful for A/B prompt comparisons. > > ```bash theme={null} theme={null} > npx @evalgate/sdk judge compare \ > --config-id 42 \ > --input "Cancel my subscription" \ > --output-a "I've canceled your plan effective today." \ > --output-b "Please visit billing settings to make changes." > ``` ## Judge credibility config Configure judge credibility thresholds and failure-mode alerts in `evalgate.config.json` at the root of your project: ```json theme={null} theme={null} { "judge": { "bootstrapSeed": 42, "tprMin": 0.70, "tnrMin": 0.70, "minLabeledSamples": 30 }, "failureModeAlerts": { "modes": { "hallucination": { "weight": 1.5, "maxPercent": 10 }, "off_topic": { "weight": 1.0, "maxPercent": 20, "maxCount": 5 }, "wrong_format": { "weight": 0.8, "maxPercent": 15 } } } } ``` > [!TIP] > > Set `bootstrapSeed` to a fixed value (for example, `42`) to make judge credibility calculations deterministic across CI runs. Without a fixed seed, bootstrap confidence intervals may vary slightly between runs. When a judge's discriminative power (TPR + TNR − 1) falls at or below 0.05, the gate skips score correction and exits with code `8` (WARN) instead of using a potentially biased score. When labeled sample count is below `minLabeledSamples`, bootstrap confidence intervals are also skipped — both conditions emit reason codes into the `judgeCredibility` block of the JSON report. ## Complete command index Every registered EvalGate CLI command. Run `npx @evalgate/sdk --help` for the same text in your terminal. > [accordion] evalgate analyze — Analyze labeled golden dataset failure modes (first pass) > > ```text theme={null} theme={null} > evalgate analyze — Analyze labeled golden dataset failure modes (first pass) > > Usage: > evalgate analyze [options] > > Options: > --dataset Labeled JSONL dataset path (default: .evalgate/golden/labeled.jsonl) > --format Output format: human (default), json > --top Number of top failure modes to show (default: 5) > ``` > [accordion] evalgate api - call any public operation from the verified OpenAPI contract > > ```text theme={null} theme={null} > evalgate api - call any public operation from the verified OpenAPI contract > > Usage: > evalgate api [options] > evalgate api --list-operations [--json] > > Options: > --path Path parameter; repeat for multiple values > --query Query parameter; repeat for multiple values > --header Declared operation header; repeat as needed > --body JSON request body or @path to a JSON file > --idempotency-key Send Idempotency-Key for a mutation > --paginate Fetch every declared cursor/offset/page response > --limit Page size when the operation declares limit > --max-pages Pagination safety bound (default: 100) > --timeout-ms Request timeout in milliseconds > --api-key API key (or EVALGATE_API_KEY) > --base-url API base URL (or EVALGATE_BASE_URL) > --organization-id Organization context (or EVALGATE_ORGANIZATION_ID) > --json Compact machine-readable JSON on stdout > > Exit codes: 0 success, 4 API/network, 5 invalid arguments, 11 local IO, 99 internal, 130 cancelled. > ``` > [accordion] evalgate auto — Plan, run, or daemonize budget-aware experiment iterations > > ```text theme={null} theme={null} > evalgate auto — Plan, run, or daemonize budget-aware experiment iterations > > Usage: > evalgate auto [options] > evalgate auto contract [--intent ] [--model ]... [--reset] > evalgate auto freeze [--branch ] > evalgate auto report [--output ] > evalgate auto export [--output ] > evalgate auto propose [--push] [--open-pr] > evalgate auto dispatch [--push] [--open-pr] [--max-runtime-minutes ] [--state-remote ] [-- ] > evalgate auto dispatch --status | --cancel [--state-remote ] > evalgate auto review [--approve] [--accept-risk ] > evalgate auto run [options] > evalgate auto daemon [options] > evalgate auto history [options] > > Options: > --objective Target failure mode or experiment goal (required) > --hypothesis Human-readable candidate hypothesis > --base Baseline run report reference (default: baseline) > --head Candidate run report path to evaluate > --budget Planned iteration budget (default: 3) > --dry-run Produce a plan without making a keep/discard decision > --output Auto report JSON path (default: .evalgate/auto/latest.json) > --format Output format: human (default), json > --cycles Daemon mode cycle count (daemon subcommand) > --interval-ms Daemon mode wait interval between cycles > --approve review: approve the shown plan (records who, when, plan hash) > --accept-risk review: accept one high-risk item by name (repeatable) > --intent contract: compile stated intent into the Optimization Contract > --model contract: a candidate target model for "try cheaper models" (repeatable) > --reset contract: drop stated intent; program.md alone applies > > Cases are split by source family into development (the optimizer sees > failures), selection (scores decide keep/discard; a regression vetoes) and > confirmation (never run by the loop). With review.required in program.md, > run and daemon refuse until the current plan is approved. Every run is held > to an Optimization Contract (stated intent or program.md): only editable > prompts change, graders, cases, baseline, policy and budgets are immutable, > confirmation cases are never run, and only a human adopts or merges. > > freeze commits the kept candidate to an isolated branch and restores your > working tree; report and export describe it for review and for EvalGate to > confirm on the reserved cases; propose prints a draft PR. Nothing merges. > > dispatch is for schedulers: it runs the approved plan in an isolated > worktree of the committed base, or refuses (exit 3) when the approval is > missing or stale (plan changed, or older than --approval-max-age-days, > default 14), the base is dirty, another dispatch is running, or the > --max-per-approval cap (default 3) is reached. The same approval and commit > are dispatched once; --max-runtime-minutes (default 120) stops a run; > --cancel stops the running dispatch and every process it started. In CI, > where each run is a fresh checkout, keep dispatch state on the remote with > --state-remote origin (refused otherwise; --local-state only for a > workspace that persists). A kept candidate is frozen with a report and > export; --open-pr opens a draft PR. Nothing merges. > ``` > [accordion] evalgate baseline — Manage regression and repository baselines > > ```text theme={null} theme={null} > evalgate baseline — Manage regression and repository baselines > > Usage: > evalgate baseline init Create starter evals/baseline.json > evalgate baseline update Run tests and update baseline > evalgate baseline preview --repository --scan --plan-hash > evalgate baseline inspect|report|run --repository --baseline > evalgate baseline create-regression|create-fix-proposal|add-to-ci --repository --baseline > ``` > [accordion] evalgate capabilities - Navigate EvalGate by product outcome > > ```text theme={null} theme={null} > evalgate capabilities - Navigate EvalGate by product outcome > > Usage: > evalgate capabilities > evalgate capabilities > evalgate capabilities [goal] --format json > > Options: > --format Output format: human (default), json > --json Alias for --format json > > The versioned capability contract maps each governed workflow to its web route, > native TypeScript/Python commands, public operation IDs, artifacts, and next step. > ``` > [accordion] evalgate check — CI/CD evaluation gate (API-based) > > ```text theme={null} theme={null} > evalgate check — CI/CD evaluation gate (API-based) > > Usage: > evalgate check [options] > > Options: > --evaluation-id Evaluation to gate on > --api-key API key (or EVALGATE_API_KEY env) > --base-url API base URL > --format Output format: human (default), json, github > --explain Show score breakdown > --min-score Fail if score < n > --max-drop Fail if score dropped > n > --policy Enforce policy (HIPAA, SOC2, etc.) > --dry-run Print the report without blocking (exit 0) > > Examples: > evalgate check --min-score 92 --evaluation-id 42 > ``` > [accordion] evalgate ci — One-command CI loop (manifest → impact → run → diff) > > ```text theme={null} theme={null} > evalgate ci — One-command CI loop (manifest → impact → run → diff) > > Usage: > evalgate ci [options] > > Options: > --base Base reference for diff > --impacted-only Run only impacted specs > --format Output format: human (default), json, github > --write-results Write run results > ``` > [accordion] evalgate cluster — Group similar traces for faster cluster-level review > > ```text theme={null} theme={null} > evalgate cluster — Group similar traces for faster cluster-level review > > Usage: > evalgate cluster [options] > > Options: > --run Run result JSON to cluster (default: searches latest run) > --clusters Requested number of clusters (default: auto) > --include-passed Include passing traces instead of only failures > --output Write cluster report JSON to disk > --format Output format: human (default), json > ``` > [accordion] evalgate compare — Side-by-side result file comparison > > ```text theme={null} theme={null} > evalgate compare — Side-by-side result file comparison > > Compares two or more saved run result JSON files. Does NOT re-run anything. > You run each model/config separately (evalgate run --write-results), then compare the artifacts. > > Usage: > evalgate compare --base --head [options] > evalgate compare --runs [file3...] [options] > > Options: > --base Baseline run result JSON file > --head Head run result JSON file > --runs N-way compare (3+ run result JSON files) > --labels Optional cosmetic labels for the output table (e.g., model names) > --format Output format: human (default), json > --sort-by Sort by: name (default), score, duration > > Examples: > evalgate compare --base .evalgate/runs/run-a.json --head .evalgate/runs/run-b.json > evalgate compare --base gpt4o.json --head claude.json --labels "GPT-4o" "Claude 3.5" > evalgate compare --runs run-a.json run-b.json run-c.json > ``` > [accordion] evalgate connect - Install EvalGate into a coding client > > ```text theme={null} theme={null} > evalgate connect - Install EvalGate into a coding client > > Usage: > evalgate connect --client [options] > > Options: > --apply Write client files; default is a non-mutating plan > --client Client to install (required) > --format human (default) or json > > This runs after signed-in repository Setup and does not start a collector. > ``` > [accordion] evalgate controls - Inspect enterprise guardrails, provider policy, retention, and drift alerts > > ```text theme={null} theme={null} > evalgate controls - Inspect enterprise guardrails, provider policy, retention, and drift alerts > > Usage: > evalgate controls [options] > > Options: > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > ``` > [accordion] evalgate datasets - Manage governed Dataset Hub artifacts > > ```text theme={null} theme={null} > evalgate datasets - Manage governed Dataset Hub artifacts > > Usage: > evalgate datasets list [--status active] > evalgate datasets create --name [--slug ] > evalgate datasets import --dataset --file [--import-format jsonl] > evalgate datasets version --dataset > evalgate datasets publish --dataset --version > evalgate datasets bind --evaluation --dataset --version [--mode snapshot] > evalgate datasets bindings --evaluation > > Options: > --idempotency-key Stable key for safe retries; generated when omitted > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --format human (default) or json > ``` > [accordion] evalgate diff — Compare two run reports > > ```text theme={null} theme={null} > evalgate diff — Compare two run reports > > Usage: > evalgate diff [options] > > Options: > --base Base branch or report path > --head Head report path > --format Output format: human (default), json > ``` > [accordion] evalgate discover — Discover behavioral specs > > ```text theme={null} theme={null} > evalgate discover — Discover behavioral specs > > Usage: > evalgate discover [options] > > Options: > --manifest Generate evaluation manifest for incremental analysis > > Also reports suite diversity and potentially redundant spec pairs. > ``` > [accordion] evalgate doctor — Comprehensive CI/CD readiness checklist > > ```text theme={null} theme={null} > evalgate doctor — Comprehensive CI/CD readiness checklist > > Usage: > evalgate doctor [options] > > Options: > --report Output JSON diagnostic bundle > --quick Skip live API checks (local-only mode) > --format Output format: human (default), json > --strict Treat warnings as failures > > Runs itemized pass/fail checks with exact remediation commands. > ``` > [accordion] evalgate evals — Evals-as-Code manifest workflow > > ```text theme={null} theme={null} > evalgate evals — Evals-as-Code manifest workflow > > Usage: > evalgate evals validate -f evalgate.yml > evalgate evals plan -f evalgate.yml > evalgate evals apply -f evalgate.yml > evalgate evals diff -f evalgate.yml > evalgate evals gate -f evalgate.yml > > Options: > -f, --file Manifest path (default: evalgate.yml) > --format human (default) or json > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --evaluationId Gate: evaluation id in manifest > --runId Gate: optional run id > ``` > [accordion] evalgate execution-state - Inspect and persist execution-state sessions > > ```text theme={null} theme={null} > evalgate execution-state - Inspect and persist execution-state sessions > > Usage: > evalgate execution-state sessions list [options] > evalgate execution-state sessions get --session [options] > evalgate execution-state sessions create --work-item-id --work-item-type --privacy-policy-id --retention-policy-id > evalgate execution-state events list --session [options] > evalgate execution-state events append --session --type --actor --payload --payload-hash --idempotency-key > evalgate execution-state snapshots list|latest --session [options] > evalgate execution-state snapshots get --snapshot [options] > evalgate execution-state snapshots reduce --session --expected-revision [options] > evalgate execution-state handoffs begin --session --idempotency-key --source-harness-ref --destination-harness-ref --destination --context-view-id --privacy-view-id > evalgate execution-state handoffs resume --bundle --acknowledgements > > Options: > --format human (default) or json > --api-key API key (or EVALGATE_API_KEY) > --base-url API base URL (or EVALGATE_BASE_URL) > --organization-id Organization context > --body Supply the complete request body for create/append/reduce > --limit Page size for list operations > --offset Offset for list operations > --paginate Fetch every declared offset page > --json Alias for --format json > ``` > [accordion] evalgate explain — Explain last gate/check failure > > ```text theme={null} theme={null} > evalgate explain — Explain last gate/check failure > > Usage: > evalgate explain [options] > > Options: > --report Path to report JSON (default: evals/regression-report.json) > --format Output format: human (default), json > ``` > [accordion] evalgate failure-modes — View and configure failure-mode taxonomy > > ```text theme={null} theme={null} > evalgate failure-modes — View and configure failure-mode taxonomy > > Usage: > evalgate failure-modes [options] > > Options: > --init Seed standard failure modes into evalgate.config.json > --add Add a custom failure mode to config alerts > --json Machine-readable output > ``` > [accordion] evalgate gate — Run the regression gate > > ```text theme={null} theme={null} > evalgate gate — Run the regression gate > > Usage: > evalgate gate [options] > > Options: > --format Output format: human (default), json, github > --offline Run deterministic local checks (default) > --allow-network Allow provider-backed evaluators and require provider evidence > --provider-required-now Require current provider evidence explicitly > --no-fresh-cache Reject provider cache reuse > --deferred-allowed Make provider unavailability inconclusive (exit 14) > --dry-run Run checks but always exit 0 (preview mode) > > $\{renderExitCodeCliHelp()\} > > Examples: > evalgate gate > evalgate gate --format json > evalgate gate --dry-run > ``` > [accordion] evalgate generate - Generate test cases via LLM > > ```text theme={null} theme={null} > evalgate generate - Generate test cases via LLM > > Usage: > evalgate generate --scenario [options] > > Options: > --scenario Scenario description (required) > --count Number of cases to generate (default: 5) > --provider LLM provider: openai or anthropic (default: openai) > --model Model name (default: gpt-4o-mini) > --output Output JSONL path (default: .evalgate/golden/generated.jsonl) > --format Output format: human (default), json > > Environment: > OPENAI_API_KEY Required when --provider openai > ANTHROPIC_API_KEY Required when --provider anthropic > > Uses an LLM to generate diverse, realistic test cases from a scenario > description. Cases are output as JSONL, ready for review. > ``` > [accordion] evalgate harness — Optimize a coding-agent harness from local metrics > > ```text theme={null} theme={null} > evalgate harness — Optimize a coding-agent harness from local metrics > > Usage: > evalgate harness adapters [--format human|json] > evalgate harness capture --adapter [--input ] > evalgate harness capture --source [--input ] > evalgate harness record --adapter --input - > evalgate harness sync [--input ] [--api-key ] [--base-url ] > evalgate harness optimize [options] > evalgate harness feedback --report --recommendation --decision > evalgate harness mutate --report --recommendation [--skill ] [--apply] > evalgate harness rollback --mutation <.evalgate/harness-mutations/id.json> > > Options: > --input Metrics-only JSONL sessions (default: .evalgate/harness-events.jsonl) > --output Write the optimization report JSON > --history Read local accepted/rejected/validated feedback > --format human (default), json, github > > The local loop never reads prompts, completions, tool arguments, traces, or > credentials. It proposes changes only after five outcome-labeled sessions and > requires the existing eval gate to validate a base-vs-candidate harness change. > Use \ > ``` > [accordion] evalgate health — Measure application latency and errors on demand > > ```text theme={null} theme={null} > evalgate health — Measure application latency and errors on demand > > Usage: > evalgate health [options] > > Options: > --config Check manifest (default: evalgate.application.json) > --output Write the redacted JSON report to disk > --format human (default), json, github > > Runs bounded HTTP and command checks sequentially in the current repository. > It does not require cloud credentials, keep a daemon alive, or post evidence. > Use --format github in CI to add the measured reasons to the PR summary. > ``` > [accordion] evalgate impact-analysis — Analyze impact of changes > > ```text theme={null} theme={null} > evalgate impact-analysis — Analyze impact of changes > > Usage: > evalgate impact-analysis [options] > > Options: > --base Base branch (default: main) > --changed-files Comma-separated list of changed files > --format Output format: human (default), json > ``` > [accordion] evalgate import — Inspect an external optimization import (experimental) > > ```text theme={null} theme={null} > evalgate import — Inspect an external optimization import (experimental) > > Usage: > evalgate import inspect [--format human|json] > > Reads a result produced by another tool, recomputes every score from its > per-case observations, and reports what the observations support and what > the import cannot establish. Supported format: Anthropic hillclimb/v2 > (state.json). > > An import is something EvalGate received, not something it proved: its > authority is investigation only, and it never authorizes a release. > > Local and read-only: no network, no credentials, no files written. > Exit 0: an assessment was produced (not a release decision). > Exit 5: the document was refused (unsupported, malformed, duplicate rows). > Exit 11: the file could not be read. > ``` > [accordion] evalgate improve — inspect governed improvement cycles > > ```text theme={null} theme={null} > evalgate improve — inspect governed improvement cycles > > Usage: > evalgate improve cycles [--state ] [--limit ] > evalgate improve inspect --cycle > evalgate improve hypotheses --cycle > evalgate improve candidates --cycle > evalgate improve partitions --cycle > evalgate improve confirmation --cycle [--action lineage|add|prepare|run|seal-reserve|activate-reserve] > evalgate improve events --cycle > evalgate improve diff --candidate > evalgate improve attempts --candidate > evalgate improve report --cycle > evalgate improve run --cycle [--scenario ] [--autonomy <0-5>] > evalgate improve adopt --cycle --candidate --reason ... > > Options: > --format json|human Output format (default: human) > --apiKey Overrides EVALGATE_API_KEY > --baseUrl Overrides EVALGATE_BASE_URL > > confirmation actions: lineage (default GET), add, prepare, run, seal-reserve, > activate-reserve. Run confirmation dispatches confirmation_only trials on the > frozen candidate; improve run queues a sandboxed ablation; adopt records a > human decision. Shadow, canary, promote, and rollback move live traffic and are > deliberately not available from the CLI. > > Exit codes: 0 success, 1 request or usage error, 2 not found, 3 auth. > ``` > [accordion] evalgate init - Initialize EvalGate in the current repository > > ```text theme={null} theme={null} > evalgate init - Initialize EvalGate in the current repository > > Usage: > evalgate init [options] > evalgate init --local [--preview|--apply] > > Options: > --local Initialize without authentication, a remote, or network > --preview Print the scaffold plan without writing > --apply Write the displayed scaffold in non-interactive use > --package-handler Override detection; repeat for a composed project > --template Start with a real working template (chatbot, codegen, agent, safety, rag, production-loop) > --list-templates Show all available templates > > Handlers: npm, pnpm, yarn, yarn-classic, yarn-modern, bun, deno, pip, uv, poetry, pipenv, pdm, conda, mamba, hatch, pixi, custom. > In a terminal, init previews and asks before writing. Automation remains > preview-only unless --apply is explicit. > ``` > [accordion] evalgate integrity - Inspect integrity and workflow-governance signals > > ```text theme={null} theme={null} > evalgate integrity - Inspect integrity and workflow-governance signals > > Usage: > evalgate integrity --evaluationId [options] > > Options: > --run Optional run ID filter > --refresh Recompute the snapshot before printing > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > ``` > [accordion] evalgate judge — Run, compare, or configure pluggable judges > > ```text theme={null} theme={null} > evalgate judge — Run, compare, or configure pluggable judges > > Usage: > evalgate judge configs [options] > evalgate judge create --name --promptTemplate [--judge provider:model ...] [options] > evalgate judge evaluate --configId --input --output [options] > evalgate judge compare --configId --input --outputA --outputB [options] > evalgate judge test --promptTemplate --input --output [--judge provider:model ...] [options] > > Options: > --judge Repeatable. Use provider:model or rule:exact_match > --aggregation all_pass, any_pass, weighted, primary_fallback > --provider openai, anthropic, google, local > --model Primary model when --judge is omitted > --apiKey API key > --baseUrl API base URL > --format Output format: human (default), json > ``` > [accordion] evalgate knowledge - Inspect learned workflow hints and playbook guidance > > ```text theme={null} theme={null} > evalgate knowledge - Inspect learned workflow hints and playbook guidance > > Usage: > evalgate knowledge --evaluationId [options] > > Options: > --failure-mode Restrict to one failure mode > --workflow-class Restrict to one workflow class > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > ``` > [accordion] evalgate label — Interactive trace labeling for golden dataset > > ```text theme={null} theme={null} > evalgate label — Interactive trace labeling for golden dataset > > Usage: > evalgate label [options] > > Options: > --run Run result JSON to label (default: searches evals/latest-run.json) > --cluster Cluster report JSON to label in cluster order > --output Labeled JSONL output path (default: .evalgate/golden/labeled.jsonl) > --format Output format: human (default), json > > Steps through traces from a run result or cluster report, allowing pass/fail labeling > and optional failure-mode tagging. Writes to canonical labeled.jsonl. > ``` > [accordion] evalgate link — Attach the current Git repository > > ```text theme={null} theme={null} > evalgate link — Attach the current Git repository > > Usage: > evalgate link [--repository owner/name] [--preview|--apply] [--format human|json] > > Interactive terminals preview and confirm. Non-interactive invocations remain > preview-only unless --apply is explicit. > ``` > [accordion] evalgate login — Authenticate this CLI > > ```text theme={null} theme={null} > evalgate login — Authenticate this CLI > > Usage: > evalgate login [--base-url ] [--scope ] [--format human|json] > > Starts a browser-approved device flow and stores the CLI session locally without > printing it. Next: evalgate link > > Options: > --scope Space-separated consented scopes. Defaults omit traces:read; > request it explicitly when collector traces must be listed. > ``` > [accordion] evalgate logout — Revoke and remove the CLI session > > ```text theme={null} theme={null} > evalgate logout — Revoke and remove the CLI session > > Usage: > evalgate logout [--format human|json] > > Revokes the saved CLI session remotely, then removes its local handle. > ``` > [accordion] evalgate mcp - Inspect, audit, or proxy MCP servers with evidence capture > > ```text theme={null} theme={null} > evalgate mcp - Inspect, audit, or proxy MCP servers with evidence capture > > Usage: > evalgate mcp inspect --manifest > evalgate mcp audit --connector --manifest > evalgate mcp proxy --runtime --server -- [args...] > > Options: > --local Do not post evidence to EvalGate > --idempotency-key Stable key for proxy session retries > --evidence-status Write machine-readable audit/proxy status JSON > --strict-persistence Exit nonzero if evidence persistence fails or conflicts > --fail-on Audit threshold: low, medium, high, critical (default: critical) > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > ``` > [accordion] evalgate measurement - Inspect and execute evaluator validation > > ```text theme={null} theme={null} > evalgate measurement - Inspect and execute evaluator validation > > Usage: > evalgate measurement runs [--status completed] > evalgate measurement releases > evalgate measurement show --id > evalgate measurement metrics --id > evalgate measurement execute --id > > Local labeled shadow pilot (repo workspace; activation blocked): > TYPESAFE_API_KEY=… pnpm measurement:shadow-pilot > TYPESAFE_API_KEY=… pnpm measurement:shadow-pilot -- --repeat 2 --format human > > Options: > --limit Result limit (1-100) > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --format human (default) or json > ``` > [accordion] evalgate migrate — Migrate legacy config formats > > ```text theme={null} theme={null} > evalgate migrate — Migrate legacy config formats > > Usage: > evalgate migrate config --in --out [options] > > Options: > --in, -i Input config path > --out, -o Output DSL path > --verbose, -v Verbose migration logs > --no-helpers Omit helper specs from output > --no-preserve-ids Do not preserve spec IDs > --no-provenance Omit provenance metadata > --dry-run Print to stdout instead of writing file > ``` > [accordion] evalgate open — Open the EvalGate control plane > > ```text theme={null} theme={null} > evalgate open — Open the EvalGate control plane > > Usage: > evalgate open [dashboard|traces|gate|evals] [--print] [--format json] > ``` > [accordion] evalgate packs - Discover and install first-party domain evaluation packs > > ```text theme={null} theme={null} > evalgate packs - Discover and install first-party domain evaluation packs > > Usage: > evalgate packs list [--domain healthcare] > evalgate packs show > evalgate packs install [--name ] > evalgate packs installations > evalgate packs installation > > Options: > --idempotency-key Stable key for safe installation retries > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --format human (default) or json > ``` > [accordion] evalgate plan - Preview an autonomous workflow plan > > ```text theme={null} theme={null} > evalgate plan - Preview an autonomous workflow plan > > Usage: > evalgate plan --objective --target-path --strategy-tracks [options] > > Options: > --target-file Read target content from disk instead of inline text > --target-content Inline target content > --iteration Planner iteration (default: 1) > --hypothesis Optional experiment hypothesis > --workflow-class Optional workflow class override > --autonomy-level Optional autonomy override > --approval-mode Optional approval mode override > --memory-mode Optional memory mode override > --background-execution Mark the plan as background-capable > --required-hooks Comma-separated required hooks > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > ``` > [accordion] evalgate playground - Author governed Playground cases > > ```text theme={null} theme={null} > evalgate playground - Author governed Playground cases > > Usage: > evalgate playground import --evaluation --file [--import-format jsonl] > evalgate playground generate --evaluation --scenario [--count 5] > evalgate playground from-trace --evaluation --trace [--failure-span ] > > Options: > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --format human (default) or json > ``` > [accordion] evalgate print-config — Show resolved config > > ```text theme={null} theme={null} > evalgate print-config — Show resolved config > > Usage: > evalgate print-config [options] > > Options: > --format Output format: human (default), json > ``` > [accordion] evalgate promote — Promote candidate eval cases to regression suite > > ```text theme={null} theme={null} > evalgate promote — Promote candidate eval cases to regression suite > > Usage: > evalgate promote Promote a specific candidate (cloud) > evalgate promote --auto Auto-promote all eligible (cloud) > evalgate promote --list List promotable candidates (cloud) > evalgate promote --local Promote local JSONL to active_gate (no API key) > > Options: > --evaluation-id Target evaluation (default: golden regression) > --apiKey API key > --baseUrl API base URL > --dry-run Show what would be promoted without making changes > --local Local filesystem promote to .evalgate/golden/active.jsonl > --input Source JSONL for --local > --output Destination JSONL for --local > --case-id Limit --local promotion to specific case ids > --require-standards Require standardsRefs on promoted cases > ``` > [accordion] evalgate prompts - Manage governed Prompt Hub artifacts > > ```text theme={null} theme={null} > evalgate prompts - Manage governed Prompt Hub artifacts > > Usage: > evalgate prompts list [--status active] > evalgate prompts create --key --name --definition > evalgate prompts show --prompt > evalgate prompts version --prompt --definition > evalgate prompts review --prompt --version --revision > evalgate prompts approve --prompt --version --revision > evalgate prompts publish --prompt --version --environment --binding-revision > evalgate prompts resolve --prompt --environment > evalgate prompts rollback --prompt --environment --binding-version --binding-revision > > Options: > --idempotency-key Stable key for safe retries; generated when omitted > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --format human (default) or json > ``` > [accordion] evalgate quality-profile - Scaffold evalgate.quality.json from product understanding > > ```text theme={null} theme={null} > evalgate quality-profile - Scaffold evalgate.quality.json from product understanding > > Usage: > evalgate quality-profile [options] > > Options: > --depth deep (default) or quick > --deep Alias for --depth deep > --apply Write the reviewed profile; preview writes nothing > --output Profile path (default: evalgate.quality.json) > --root Project directory (default: current directory) > --format human (default) or json > > Preview-by-default. Does not synthesize cases, call a model, or accept a > baseline. If docs already contain contrastive examples, they are copied into > the profile. Otherwise the file is written with empty example slots for the > builder to fill before \ > ``` > [accordion] evalgate red-team — Fetch governed release and finding evidence > > ```text theme={null} theme={null} > evalgate red-team — Fetch governed release and finding evidence > > Usage: > evalgate red-team evidence --run [options] > evalgate red-team finding --finding [options] > > Options: > --run Durable Red Team campaign run (required) > --finding Durable Red Team finding (required for finding) > --output Artifact path (default: .evalgate/red-team-evidence.json) > --api-key API key (or EVALGATE_API_KEY) > --base-url API base URL > --organization-id Organization header override > --format human|json Output format (default: human) > > Trust requirements: > Strict evalgate.config.json policies must pin redTeam.expectedTarget, > expectedOrganizationId, expectedSnapshotHash, and the > expectedAttestationPublicKeySha256 trust anchor; expectedAttestationKeyId is > optional. The file is written atomically only after schema, canonical hash, > Ed25519 compiler attestation, organization, snapshot, and exact attacked-target > identity all verify. Machine failures use policy, API, argument, I/O, or internal > exit categories instead of collapsing to BAD_ARGS. > ``` > [accordion] evalgate replay — Replay a candidate eval case (deprecated alias) > > ```text theme={null} theme={null} > evalgate replay — Replay a candidate eval case (deprecated alias) > > Deprecated: use \ > ``` > [accordion] evalgate replay-candidate — Replay a candidate eval case > > ```text theme={null} theme={null} > evalgate replay-candidate — Replay a candidate eval case > > Usage: > evalgate replay-candidate > > Options: > --model Override model > --format Output format: human (default), json > --apiKey API key > --baseUrl API base URL > ``` > [accordion] evalgate replay-decision - Compare two run reports and decide keep/discard > > ```text theme={null} theme={null} > evalgate replay-decision - Compare two run reports and decide keep/discard > > Usage: > evalgate replay-decision --previous --current [options] > > Options: > --format Output format: human (default), json > > Requires normalizedBudget in evalgate.config.json. > ``` > [accordion] evalgate repo - Scan an exact connected repository commit for AI systems > > ```text theme={null} theme={null} > evalgate repo - Scan an exact connected repository commit for AI systems > > Usage: > evalgate repo repositories > evalgate repo scan --repository [--head-sha ] > evalgate repo scans --repository > evalgate repo show --repository --scan > evalgate repo ask --repository --question [--head-sha ] > > Options: > --repository Connected repository identifier > --scan Existing scan identifier for show > --question Question for repo ask > --head-sha Exact 40-character commit SHA (scan and ask) > --idempotency-key Stable key for safe scan retries > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --format human (default) or json > ``` > [accordion] evalgate rollback - Remove files created by one init transaction > > ```text theme={null} theme={null} > evalgate rollback - Remove files created by one init transaction > > Usage: > evalgate rollback --transaction > evalgate rollback > > Options: > --transaction Transaction recorded in evalgate.project.json > > Rollback removes only transaction-owned files that have not been edited since > init. Pre-existing and subsequently edited files are preserved. > ``` > [accordion] evalgate run — Run evaluation specifications > > ```text theme={null} theme={null} > evalgate run — Run evaluation specifications > > Usage: > evalgate run [options] > > Options: > --spec-ids Comma-separated list of spec IDs > --impacted-only Run only impacted specs (requires --base) > --base Base branch for impact analysis > --format Output format: human (default), json > --write-results Write results to .evalgate/last-run.json > ``` > [accordion] evalgate run-cli - Run an agent/CLI command with evidence capture > > ```text theme={null} theme={null} > evalgate run-cli - Run an agent/CLI command with evidence capture > > Usage: > evalgate run-cli --runtime [--permission-profile ] -- [args...] > > Options: > --runtime Agent runtime key > --permission-profile Permission profile key > --idempotency-key Stable key for safe retries > --evidence-status Write machine-readable persistence status JSON > --local Capture locally without posting to EvalGate > --strict-persistence Exit nonzero if EvalGate persistence fails or conflicts > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > ``` > [accordion] evalgate serve — run your feature for EvalGate evaluations > > ```text theme={null} theme={null} > evalgate serve — run your feature for EvalGate evaluations > > Usage: > evalgate serve [options] -- [args...] > > Each evaluation case runs once. The case arrives as JSON on stdin: > \{"prompt", "metadata", "evaluationRunId", "testCaseId", "job", "target"\} > and EVALGATE_TARGET_MODEL / EVALGATE_TARGET_PROVIDER / EVALGATE_TARGET_EFFORT > are set when given. Print the answer on stdout, either as plain text or as > \{"output": "...", "execution": \{"mode": "live", "served": \{...\}, "usage": \{...\}\}\} > so EvalGate can record the served model, token usage and billed cost. > A non-zero exit fails the case; stderr is streamed as logs. > > Runner identity: > --register Register this machine as a runner (needs an admin API key) > --name Runner name for --register (default: hostname) > --runner-id Existing runner (or EVALGATE_RUNNER_ID) > EVALGATE_RUNNER_BOOTSTRAP_TOKEN Its bootstrap token (rrb_…); keep it out of shell history > > Target (registers or reuses an immutable, content-hashed target): > --target Target key, e.g. support-answerer > --entrypoint Repository path of the feature, e.g. src/answer.ts > --kind prompt | scorer | tool | workflow | agent (default: agent) > --environment dev | staging | prod (default: dev) > --model Model the feature runs with (recorded as requested) > --provider Its provider > --effort Reasoning effort or equivalent setting > --allow-origin Network origin the feature needs (repeatable) > --wall-ms Per-case wall-clock budget (default: 120000) > > Execution reporting: > --mode live|simulated How the command runs when it does not say so itself > --target-version Code version (default: git HEAD, "-dirty" if modified) > > Loop: > --once Execute at most one case, then exit > --max-jobs Stop after n cases > --poll-interval Idle poll interval (default: 2000) > --format human|jsonl Output format (default: human) > --base-url EvalGate URL (or EVALGATE_BASE_URL) > --api-key API key for --register/--target (or EVALGATE_API_KEY) > > ``` > [accordion] evalgate share — Create share link for a run > > ```text theme={null} theme={null} > evalgate share — Create share link for a run > > Usage: > evalgate share [options] > > Options: > --scope Share scope > --evaluationId Evaluation ID > --runId Run ID > --expires Expiry duration (e.g. 7d) > --apiKey API key > --format Output format: human (default), json > ``` > [accordion] evalgate status — Summarize activation and gate readiness > > ```text theme={null} theme={null} > evalgate status — Summarize activation and gate readiness > > Usage: > evalgate status [--format human|json] > ``` > [accordion] evalgate sync — Push/pull labels and prompt state between CLI and web > > ```text theme={null} theme={null} > evalgate sync — Push/pull labels and prompt state between CLI and web > > Usage: > evalgate sync [options] > > Options: > --project-key Project key slug (from evalgate.config.json or --project-key) > --evaluation-id Direct evaluation ID (bypasses project-key resolution) > --prompt-id Prompt ID to pull or push prompt versions for > --target-path Local path for prompt pull/push > --labeled-dataset JSONL file of labeled cases to push or merge into > --pull-labels Pull web-labeled cases into the local JSONL file > --push-prompt Push target-path content as a web prompt version > --pull-prompt Pull active prompt content when also pushing > --expected-prompt-active-version-id Expected active prompt version for stale-write protection > --dry-run Show what would happen without making changes > --format Output format: human (default), json > > Examples: > evalgate sync --project-key my-chatbot > evalgate sync --project-key my-chatbot --labeled-dataset .evalgate/golden/labeled.jsonl --pull-labels > evalgate sync --evaluation-id 42 --prompt-id prompt_1 --target-path prompts/system.md --push-prompt > evalgate sync --dry-run > ``` > [accordion] evalgate synthesize — Draft quarantined eval cases from examples or labeled failures > > ```text theme={null} theme={null} > evalgate synthesize — Draft quarantined eval cases from examples or labeled failures > > Usage: > evalgate synthesize --profile [--apply] > evalgate synthesize [options] > > Options: > --profile Examples-first quality profile JSON; previews without writing by default > --apply Write reviewed profile drafts after preview > --dataset Labeled JSONL dataset path (default: .evalgate/golden/labeled.jsonl) > --dimensions Dimension matrix JSON path > --failure-mode Restrict to one or more failure modes (comma-separated) > --kind Synthesis kind: dimension (default), paraphrase, harder_variant > --count Number of synthetic cases to generate (default: mode × dimension coverage) > --output Synthetic JSONL output path (default: .evalgate/golden/synthetic.jsonl) > --format Output format: human (default), json > > Profile mode is deterministic and local: preview makes no writes, network calls, > model calls, Git mutations, baseline changes, or promotions. > ``` > [accordion] evalgate synthesize-run — Run durable, reviewable synthetic-data generation > > ```text theme={null} theme={null} > evalgate synthesize-run — Run durable, reviewable synthetic-data generation > > Usage: > evalgate synthesize-run start --dataset [options] > evalgate synthesize-run status --run [options] > evalgate synthesize-run review --run --item --decision accepted|rejected > evalgate synthesize-run retry|cancel|publish --run [options] > > Options: > --dimensions Dimension matrix JSON > --evaluation Evaluation to associate with the run > --dataset-id Dataset to extend on publish > --count Requested generated cases (1-1000) > --failure-mode Comma-separated failure modes > --stage-mission-proposals Create reviewable mission proposals on publish > --publish-dataset-version Create an immutable Dataset Hub version on publish > --idempotency-key Stable key for safe retries > --apiKey API key (or EVALGATE_API_KEY) > --baseUrl API base URL > --format human (default) or json > > Start is asynchronous and durable. Status output always includes the next command. > ``` > [accordion] evalgate templates — List starter evaluation templates > > ```text theme={null} theme={null} > evalgate templates — List starter evaluation templates > > Usage: > evalgate templates > > Prints the same catalog as evalgate init --list-templates. > ``` > [accordion] evalgate trace — Inspect stored trajectory metrics for a run > > ```text theme={null} theme={null} > evalgate trace — Inspect stored trajectory metrics for a run > > Usage: > evalgate trace --evaluationId --run [options] > > Options: > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > --case Restrict output to one test case > > --run expects an evaluation run ID from /api/evaluations/\{id\}/runs. > A collector trace_db_id is a different identifier and will 404 here. > ``` > [accordion] evalgate understand - Build an evidence-backed working theory of the product > > ```text theme={null} theme={null} > evalgate understand - Build an evidence-backed working theory of the product > > Usage: > evalgate understand [options] > > Options: > --depth deep (default) or quick > --deep Alias for --depth deep > --apply Write the reviewed context; preview writes nothing > --output Committed context path (default: evalgate.product.json) > --root Project or product directory (default: current directory) > --name Confirm the product name > --description Confirm what the product does > --audience Confirm who receives the result > --job Confirm the user's primary job > --primary-output Confirm the result the AI delivers > --format human (default), json, or typed jsonl progress > --interpret Also ask a model to interpret the scanned evidence > (needs ANTHROPIC_API_KEY or OPENAI_API_KEY; spends tokens) > --model Model for --interpret > --reunderstand Ignore the cached interpretation and ask again > > Default deep mode reads high-signal product files across the repository and > returns a provisional theory, evidence, confidence, product-specific risk > hypotheses, and any contrastive examples found in docs. Quick mode keeps the > 14-file first-read budget. After repository installation it runs without > network, model calls, project commands, Git inspection, baseline changes, > promotion, or telemetry setup. Generated risks are hypotheses until a person > confirms them or real behavior provides evidence. > > --interpret sends only the scanned excerpts to the model and returns AI > features, their entrypoints and objectives as hypotheses with falsifiers. > Proposals citing files the scan did not read are dropped. Results are cached > under .evalgate/cache/understand by the scan's content hash, analyzer version, > model and settings, so an unchanged repository is not interpreted twice. > Interpretation never ranks features by cost; cost comes from measured runs. > ``` > [accordion] evalgate validate — Validate spec files without running them > > ```text theme={null} theme={null} > evalgate validate — Validate spec files without running them > > Usage: > evalgate validate [options] > > Options: > --format Output format: human (default), json > ``` > [accordion] evalgate watch — Watch mode (re-execute on file save) > > ```text theme={null} theme={null} > evalgate watch — Watch mode (re-execute on file save) > > Usage: > evalgate run --watch [options] > evalgate watch [options] > > Options: > --debounce Debounce interval (default: 300ms) > --no-clear Don't clear screen between runs > --format Output format: human (default), json > --write-results Write results to .evalgate/last-run.json > > Examples: > evalgate run --watch > evalgate watch --write-results > ``` > [accordion] evalgate whoami — Show the authenticated identity > > ```text theme={null} theme={null} > evalgate whoami — Show the authenticated identity > > Usage: > evalgate whoami [--format human|json] > ``` > [accordion] evalgate workflow - Inspect or generate workflow-native coverage bundles > > ```text theme={null} theme={null} > evalgate workflow - Inspect or generate workflow-native coverage bundles > > Usage: > evalgate workflow --evaluationId [options] > > Options: > --synthesize Generate workflow cases and experiment recipes > --run Optional source run for synthesis > --case-count Optional generated case count (max 20) > --include-passed Include passed runs when synthesizing > --persist-candidates Persist generated candidate cases > --persist-artifacts Persist workflow artifacts > --apiKey API key (or EVALGATE_API_KEY env) > --baseUrl API base URL > --format Output format: human (default), json > ``` ### Latency enforcement in SDK 3.13.1 SDK 3.13.1 supports `tolerance.latencyEnforcement` in an accepted local baseline. Upgrade older installed versions before relying on this policy. ```json theme={null} { "tolerance": { "latencyEnforcement": "advisory", "maxLatencyIncreaseMs": 200 } } ``` `advisory` is the default for local and shared-CI evaluations. A measured increase above the tolerance appears as **WARN**, with the baseline, current measurement and delta retained. It cannot block solely because of latency. For a dedicated performance benchmark, explicitly choose `blocking` and a finite nonnegative `maxLatencyIncreaseMs`. This is a permitted increase from baseline, not an absolute response-time limit. Missing required comparison measurements block instead of appearing unchanged. Execution timeouts, unregistered coverage, incomplete execution evidence and behavioral failures remain blocking under both policies. Change baseline policy through your reviewed baseline workflow; editing an accepted checksummed baseline without reacceptance fails its integrity check. A performance policy is not a deployment permission or baseline approval. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.