Set up CI/CD regression gates with EvalGate

Start with gate. Complete the five-minute loop once, then run that same command on every pull request. Add the later modes only when you need their different evidence contract.

Mode	What it proves	Credentials and cost	What a red build means	Artifacts
First: `gate`	Reviewed local checks still pass against the accepted baseline	No EvalGate credential or result charge; deterministic cases need no provider spend	Regression (exit `2`), or a distinct configuration, evidence or infrastructure failure; inspect the reason	`.evalgate/last-report.json`, `evals/regression-report.json`
Later: `ci`	Discovered specs execute, with optional base/head comparison	No EvalGate credential; opted-in model execution can incur BYOK spend	Spec execution, coverage or comparison failed; not the same baseline decision as `gate`	`.evalgate/` run and diff artifacts with `--write-results`
Later: `check`	Hosted evidence satisfies configured case, component, provider, judge and policy requirements	Organization credential and hosted usage; model-backed evidence adds provider spend	Required evidence is missing, invalid or below a threshold; a high blended score cannot override it	Hosted check report and optional failure import

The quality scoring contract describes components, missing measurements and evidence completeness. The blended score is descriptive, not a standalone release decision. Cost thresholds on completed evidence are not pre-execution spending limits.

Local gate setup

Complete the five-minute local gate first. It covers preview, explicit apply, replacement of every placeholder, discovery, the reviewed-source commit and acceptance of a non-empty passing baseline. CI consumes those reviewed files; it never accepts a baseline for you. Commit @evalgate/sdk as a dev dependency and its package lockfile. Use the workflow below so npm ci installs the reviewed version and npx --no-install fails if that dependency is missing. The published 3.15.0 initializer pins its own 3.15.0 in the generated workflow. The source generator derives that pin from its own SDK_VERSION, so it does not carry an independent stale version. The lockfile-only template below is an alternative for projects that commit the SDK dependency. Compare older generated workflows with your installed SDK before committing them.

.github/workflows/evalgate-gate.yml

name: EvalGate Gate
on:
  pull_request:
    branches: [main]
  push:
    branches: [main]
jobs:
  regression-gate:
    runs-on: ubuntu-latest
    permissions:
      contents: read
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '24'
          cache: npm
      - run: npm ci
      - name: EvalGate regression gate
        run: npx --no-install @evalgate/sdk gate --format github --offline
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalgate-report
          path: .evalgate/last-report.json
          if-no-files-found: error

The canonical .evalgate/last-report.json is the release decision envelope; evals/regression-report.json is the detailed regression artifact. Baselines are reviewed evidence and are never automatically updated, committed, or pushed by generated CI. Accept a baseline explicitly after reviewing a clean run with evalgate baseline update.

The built-in local gate compares test pass state and parsed test count against the baseline. If your test script runs AI evals, this gates AI behavior. Otherwise, it gates test health until you add eval specs or a custom eval:regression-gate script.

Later: hosted checks

Use check after the local loop when you need shared hosted evidence, independent judge credibility and import-on-failure behavior. Review availability and configure every required component; a score alone cannot demonstrate unmeasured safety.

.github/workflows/evalgate-platform-gate.yml

name: EvalGate Platform Gate
on:
  pull_request:
    branches: [main]
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '24'
      - run: npm ci
      - name: EvalGate quality gate
        env:
          EVALGATE_API_KEY: ${{ secrets.EVALGATE_API_KEY }}
        run: npx @evalgate/sdk check --format github --onFail import

check --format github emits GitHub annotations for failed cases and writes a step summary with the verdict, score, baseline score, delta, and top failing cases. --onFail import uploads the failed run context to the platform so you can review it in the dashboard.

GitLab CI recipe

GitLab supports a local gate recipe only. Connecting a GitLab repository for Repository Intelligence or automatic AI discovery is unsupported. This is not a native GitLab integration. Use JSON output and retain both decision artifacts:

.gitlab-ci.yml

eval-gate:
  stage: test
  image: node:24
  script:
    - npm ci
    - npx --no-install @evalgate/sdk gate --format json
  artifacts:
    when: always
    paths:
      - evals/regression-report.json
      - .evalgate/last-report.json
  only:
    - merge_requests
    - main

Setting quality thresholds

Define local gate thresholds in evalgate.config.json. The built-in local gate uses evals/baseline.json; platform gates use the quality and judge settings configured for the evaluation.

evalgate.config.json

{
  "gate": {
    "baseline": "evals/baseline.json",
    "report": "evals/regression-report.json"
  },
  "judge": {
    "tprMin": 0.7,
    "tnrMin": 0.7,
    "minLabeledSamples": 30
  }
}

Store evalgate.config.json and evals/baseline.json in version control. Changing thresholds or baselines without review changes what your CI gate proves.

Exit codes

The process exit status is the shell/CI result. JSON reports may additionally carry effectiveExitCode and releaseReady when preview mode or evidence policy separates process completion from the release decision. The complete, version-bound contract is available from evalgate capabilities --format json.

Code	Symbol	Meaning	Release meaning
`0`	`PASS`	All required evidence passed the configured gate.	Release-ready when the report also says `releaseReady=true`.
`1`	`SCORE_BELOW`	The measured score is below the configured minimum.	Blocks release.
`2`	`REGRESSION`	Measured behavior regressed beyond the accepted baseline tolerance.	Blocks release.
`3`	`POLICY_VIOLATION`	A configured policy check failed.	Blocks release.
`4`	`API_ERROR`	The hosted API or network request failed.	Release readiness is inconclusive; retry or use an admissible cache.
`5`	`BAD_ARGS`	The command arguments are invalid.	Blocks release until configuration is corrected.
`6`	`LOW_N`	The measured sample is too small for the configured decision.	Blocks release unless policy explicitly allows weak evidence.
`7`	`WEAK_EVIDENCE`	Evidence exists but is not strong enough for the gate.	Blocks release.
`8`	`WARN_REGRESSION`	A metric entered the configured warning band.	Policy-dependent; never treat as a stronger pass than reported.
`9`	`FAILURE_MODE_THRESHOLD`	A failure-mode threshold was exceeded.	Blocks release.
`10`	`JUDGE_CREDIBILITY_UNTRUSTWORTHY`	Judge calibration is too weak to support a trusted decision.	Blocks release or remains inconclusive per policy.
`11`	`IO_ERROR`	Required report or evidence could not be persisted.	Blocks release; a missing canonical report is not a pass.
`12`	`BASELINE_TAMPER`	The baseline checksum or attestation is invalid.	Blocks release until the reviewed baseline is restored.
`13`	`PROVIDER_UNAVAILABLE`	Required provider evidence was unavailable or stale.	Inconclusive or blocked; never release-green.
`14`	`PROVIDER_INCONCLUSIVE`	Provider work was deferred under an explicit policy.	Inconclusive unless policy explicitly admits deferred evidence.
`15`	`BASELINE_NOT_ACCEPTED`	The current run is measurable, but no reviewed baseline has been accepted.	Blocks release until a reviewed baseline is accepted.
`99`	`INTERNAL_ERROR`	An unexpected internal error occurred.	Blocks release.
`130`	`CANCELLED`	The operation was cancelled by the user or orchestrator.	No release decision was made.

These meanings are authoritative for the installed runtime. Agents and CI integrations can read the complete version-bound contract with npx @evalgate/sdk capabilities --format json.

CLI commands reference

npx @evalgate/sdk gate
npx @evalgate/sdk gate --format github
npx @evalgate/sdk gate --format json

npx @evalgate/sdk ci --format github --write-results --base main
npx @evalgate/sdk ci --format github --write-results --base main --impacted-only

npx @evalgate/sdk check --format github --onFail import
npx @evalgate/sdk check --format json

Best practices

Start local

Use the local gate first so every repo gets a baseline before the team adopts platform workflows.

Upload artifacts

Keep .evalgate/ or evals/regression-report.json as CI artifacts so failures are debuggable after the job exits.

Use impacted-only deliberately

Add --impacted-only only after your spec manifest is stable. Unknown changes fall back to broader runs.

Promote reviewed cases

Treat generated candidate cases as drafts until a human or promotion workflow approves them for gating.

Later: eval artifact CI

Use ci when your repo contains EvalGate spec files and you want each pull request to write .evalgate/ run artifacts.

.github/workflows/evalgate.yml

name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
      - run: npm ci
      - run: npx @evalgate/sdk ci --format github --write-results --base main
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: evalgate-results
          path: .evalgate/

That command:

Discovers evaluation specs automatically
Runs all specs by default
Writes run artifacts when --write-results is set
Compares against the base branch when --base is set
Writes a GitHub step summary and annotations when --format github is set
Exits with the appropriate code (see exit codes)

Add --impacted-only to run only specs affected by the current diff. Impact analysis requires a base branch, so use it with --base main or another base ref.

Other CI hosts and languages

Jenkins and Buildkite can invoke the same local command and capture artifacts. JUnit XML is a source candidate until its release is listed in feature status; use gate --format junit > evalgate.xml only with a build that advertises it. This is generic CI execution, not a native integration. TypeScript and Python are evaluation runtimes. Go/Ruby generated REST clients are not runners; Java and .NET evaluation runners are unsupported.

Get Started

Core Concepts

Guides

SDK Reference

Platform

CI/CD regression gates

Set up CI/CD regression gates with EvalGate

Local gate setup

Later: hosted checks

GitLab CI recipe

Setting quality thresholds

Exit codes

CLI commands reference

Best practices

Start local

Upload artifacts

Use impacted-only deliberately

Promote reviewed cases

Later: eval artifact CI

Other CI hosts and languages

​Set up CI/CD regression gates with EvalGate

​Local gate setup

​Later: hosted checks

​GitLab CI recipe

​Setting quality thresholds

​Exit codes

​CLI commands reference

​Best practices

Start local

Upload artifacts

Use impacted-only deliberately

Promote reviewed cases

​Later: eval artifact CI

​Other CI hosts and languages

Set up CI/CD regression gates with EvalGate

Local gate setup

Later: hosted checks

GitLab CI recipe

Setting quality thresholds

Exit codes

CLI commands reference

Best practices

Later: eval artifact CI

Other CI hosts and languages