> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Python

# evalgate-sdk Python reference

> Practical reference for evalgate-sdk — generated OpenAPI access, purpose-built helpers, async usage, tracing, evaluations, judges, test suites, and CLI commands.

The `evalgate-sdk` package is the Python surface for EvalGate's evaluation control plane. Purpose-built helpers use snake\_case names. The generated OpenAPI client provides structural access to every public REST operation by stable operation ID; it does not imply that every operation has a dedicated convenience method.

`PRODUCT_CAPABILITY_CONTRACT` exposes the same versioned workflow-navigation
map as `evalgate capabilities --format json`. Agents can discover the web route,
native Python/TypeScript commands, public operations, artifacts, and next step
for each governed core workflow without scraping help text.

From contract schema 3, capabilities that span operations with different
permissions also declare those operations, and `select_capability_operation()`
resolves a stated intent plus already-demonstrated context to one supported
operation, or to a typed missing-precondition reason. Both SDKs run the same
selection vectors, so the two surfaces cannot drift apart on what a caller may
do. The context you pass is informational: it states what has been
demonstrated and never itself grants authority, unrecognised values are
ignored rather than honoured, and the named command still authorizes for
itself when it runs.

> [!NOTE]
>
> `select_capability_operation()` ships in `evalgate-sdk==3.15.0` on PyPI
> (import it from `evalgate_sdk.capability_selection`). The Python and
> TypeScript SDKs are released together at 3.15.0.

## Install

> [tab] Core SDK
>
> ```bash theme={null} theme={null}
> pip install evalgate-sdk
> ```

> [tab] SDK + CLI
>
> ```bash theme={null} theme={null}
> pip install evalgate-sdk
> ```
>
> This installs the `evalgate` CLI command alongside the SDK.

> [tab] With integrations
>
> ```bash theme={null} theme={null}
> pip install "evalgate-sdk[openai]"      # OpenAI tracing
> pip install "evalgate-sdk[anthropic]"   # Anthropic tracing
> pip install "evalgate-sdk[langchain]"   # LangChain tracing
> pip install "evalgate-sdk[all]"         # OpenAI, Anthropic, and LangChain (the CLI is always included)
> ```

> [!NOTE]
>
> PyPI serves `evalgate-sdk==3.15.0`, the same release line as
> `@evalgate/sdk@3.15.0` on npm. It includes `init --local`, `baseline`,
> `gate`, and the other CLI workflows. Check `pip show evalgate-sdk` in your
> environment, and see [feature status](https://evalgate.com/docs/platform/feature-status.md) for maturity.

For the current TypeScript and Python CLI source, `evalgate login` completes a
human-approved device flow and stores the session outside the repository in the operating-system user
configuration directory. Windows uses `%APPDATA%\evalgate\config.json`, macOS
uses `~/Library/Application Support/evalgate/config.json`, and Linux uses
`${XDG_CONFIG_HOME:-~/.config}/evalgate/config.json`; managed environments may
set an absolute `EVALGATE_CONFIG_HOME`. Repository configuration cannot provide
a bearer key or redirect a saved/environment key to another API origin.
Alternate origins require HTTPS except localhost and loopback development.

`evalgate init --local` (published on PyPI since 3.13.0) gives a cloud coding
agent a deterministic gate before any hosted setup. That command requires no
account, browser, organization, GitHub remote, API key, or network request and
creates no hosted identity.

> [!WARNING]
>
> Current source ignores the legacy repository-local `.evalgate/config.json`
> credential. Reconfigure it, remove the file, and rotate the key if it may have
> entered Git history. Check [feature status](https://evalgate.com/docs/platform/feature-status.md) before
> assuming this source behavior is present in the current PyPI package.

> [!NOTE]
>
> The canonical PyPI package name is `evalgate-sdk`. Import it as `evalgate_sdk`. If you have the legacy `pauly4010-evalgate-sdk` package installed, migrate to `evalgate-sdk`.

> [!NOTE]
>
> The CrewAI and AutoGen tracing adapters are lightweight wrappers and do not install those frameworks. Install the CrewAI or AutoGen package used by your application separately.

## Import and initialize

```python theme={null} theme={null}
from evalgate_sdk import AIEvalClient
```

> [tab] Environment variables
>
> Set `EVALGATE_API_KEY` and `EVALGATE_ORGANIZATION_ID` (and optionally `EVALGATE_BASE_URL`) in your environment, then call `init()` with no arguments:
>
> ```python theme={null} theme={null}
> client = AIEvalClient.init()  # reads EVALGATE_API_KEY and EVALGATE_ORGANIZATION_ID
> ```

> [tab] Explicit config
>
> ```python theme={null} theme={null}
> client = AIEvalClient(
>     api_key='your-api-key',
>     organization_id='00000000-0000-4000-8000-000000000001',
>     base_url='https://www.evalgate.com',  # default; or EVALGATE_BASE_URL
>     timeout=30000,
>     debug=False,
> )
> ```

## Async usage

The Python SDK is async-first. Use `asyncio.run()` for top-level scripts, or `await` inside an async function:

```python theme={null} theme={null}
import asyncio
from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams, CreateSpanParams

client = AIEvalClient.init()

async def main():
    trace = await client.traces.create(CreateTraceParams(
        name='Chat Completion',
        metadata={'model': 'gpt-5.2'},
    ))

    await client.traces.create_span(trace.id, CreateSpanParams(
        name='OpenAI API Call',
        type='llm',
        input='What is AI?',
        output='AI stands for Artificial Intelligence...',
        metadata={'tokens': 150, 'latency_ms': 1200},
    ))

asyncio.run(main())
```

## Call any public OpenAPI operation

`OpenApiClient` consumes the generated operation registry shared with the public OpenAPI contract. Its `operation_id` type is a generated `Literal` union, and the runtime validates paths, query names, headers, body requirements, pagination, and cancellation.

```python theme={null} theme={null}
import asyncio
import os

from evalgate_sdk.openapi_client import OpenApiClient

async def main():
    async with OpenApiClient(
        api_key=os.environ["EVALGATE_API_KEY"],
        organization_id=os.environ.get("EVALGATE_ORGANIZATION_ID"),
    ) as api:
        page = await api.request(
            "get_evaluations",
            query={"limit": 25, "offset": 0},
        )

        created = await api.request(
            "post_evaluations",
            body={"name": "Support quality"},
            idempotency_key="create-support-quality-v1",
        )

asyncio.run(main())
```

Paginated operations use a bounded async iterator. A cancellation event stops an in-flight request with the stable `CANCELLED` SDK error.

```python theme={null} theme={null}
cancel = asyncio.Event()

async for response in api.paginate(
    "get_evaluations",
    query={"offset": 0},
    limit=50,
    max_pages=20,
    cancel_event=cancel,
):
    process_page(response)
```

## Client methods

The purpose-built modules below use Pythonic snake\_case names. They are convenience surfaces, not the public-operation inventory; use `OpenApiClient` for contract-generated access to operations not listed here.

### Prompt Hub and Dataset Hub

```python theme={null} theme={null}
import uuid

prompt = await client.prompts.create(
    key="support-answer",
    name="Support answer",
    definition={
        "messages": [{"id": "user", "role": "user", "content": "Answer {{input}} using policy."}],
        "inputSchema": {"type": "object", "properties": {"input": {"type": "string"}}},
    },
    idempotency_key=str(uuid.uuid4()),
)
draft = prompt["version"]
reviewed = await client.prompts.submit_for_review(
    prompt["artifactId"], draft["id"],
    expected_revision=draft["revision"], idempotency_key=str(uuid.uuid4()),
)
approved = await client.prompts.approve(
    prompt["artifactId"], draft["id"],
    expected_revision=reviewed["revision"], idempotency_key=str(uuid.uuid4()),
)
await client.prompts.publish(
    prompt["artifactId"], approved["id"],
    environment="production",
    expected_binding_version_id=None,  # first publish to this environment
    expected_binding_revision=0,
    idempotency_key=str(uuid.uuid4()),
)

dataset = await client.datasets.create(name="Support regressions")
await client.datasets.import_rows(
    dataset["id"],
    format="jsonl",
    content='{"input":"cancel","expectedOutput":"policy-safe answer"}',
    idempotency_key=str(uuid.uuid4()),
)
version = await client.datasets.create_version(dataset["id"], idempotency_key=str(uuid.uuid4()))
for step in (client.datasets.review, client.datasets.approve, client.datasets.publish):
    await step(dataset["id"], version["id"], idempotency_key=str(uuid.uuid4()))
await client.datasets.bind(
    evaluation_id,
    dataset_id=dataset["id"],
    dataset_version_id=version["id"],
    idempotency_key=str(uuid.uuid4()),
)
```

The Python CLI exposes both `evalgate datasets` for the common governed
lifecycle and `evalgate api <operation-id>` for the complete generated public
operation registry.

### Evaluation packs, Playground, and repository intelligence

```python theme={null} theme={null}
repositories = await client.repository_intelligence.list_repositories()
repository_id = repositories["repositories"][0]["id"]
scan = await client.repository_intelligence.start_scan(
    repository_id,
    idempotency_key=str(uuid.uuid4()),
    head_sha="a" * 40,
)
answer = await client.repository_intelligence.answer_question(
    repository_id,
    question_text="What AI models, agents, tools, and evals exist?",
    graph_version_id=scan["scanRun"]["graphVersionId"],
)

packs = await client.eval_packs.list(domain="coding")
installed = await client.eval_packs.install(
    "coding-agent-release-safety",
    idempotency_key=str(uuid.uuid4()),
)

await client.playground.create_case_from_trace(
    evaluation_id,
    trace_id=123,
)
```

The matching native CLI workflow is `evalgate repo` followed by `evalgate packs`
after reviewing its evidence; `evalgate playground` turns selected traces into
governed cases. Repository scanning reads protected source at one exact
commit and never executes repository code; source detection is not represented
as runtime confirmation.

### Traces

```python theme={null} theme={null}
from evalgate_sdk.types import CreateSpanParams, CreateTraceParams, ListTracesParams

# Create a trace
trace = await client.traces.create(CreateTraceParams(
    name='Chat Completion',
    metadata={'model': 'gpt-5.2'},
))

# Add a span
await client.traces.create_span(trace.id, CreateSpanParams(
    name='LLM Call',
    type='llm',
    input='...',
    output='...',
))

# List traces
await client.traces.list(ListTracesParams(limit=50, status='success'))

# Get a trace with its spans
await client.traces.get(trace_id)

# Delete a trace
await client.traces.delete(trace_id)
```

### Evaluations

```python theme={null} theme={null}
from evalgate_sdk.types import CreateEvaluationParams, CreateRunParams

# Create an evaluation
evaluation = await client.evaluations.create(CreateEvaluationParams(
    name='Safety Test',
    type='unit_test',
))

# Create and inspect runs
run = await client.evaluations.create_run(
    evaluation.id,
    CreateRunParams(execution_settings={'environment': 'ci'}),
)
await client.evaluations.list_runs(evaluation.id)
await client.evaluations.get_run(evaluation.id, run.id)
```

### Prompt versions

`client.evaluations` exposes `list_prompt_versions`, `set_active_prompt_version`, and `rollback_prompt`. The same operations are available through the generated OpenAPI contract, which keeps path encoding, required parameters, and body validation on the same surface as the other public operations:

```python theme={null} theme={null}
import os

from evalgate_sdk.openapi_client import OpenApiClient

evaluation_id = 42

async with OpenApiClient(api_key=os.environ['EVALGATE_API_KEY']) as api:
    versions = await api.request(
        'get_evaluations_id_prompts_promptId_versions',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
    )
    activated = await api.request(
        'post_evaluations_id_prompts_promptId_active',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
        body={'versionId': versions['versions'][0]['id']},
    )
    rollback = await api.request(
        'post_evaluations_id_prompts_promptId_rollback',
        path={'id': evaluation_id, 'promptId': 'support/primary'},
    )
```

### LLM judge

```python theme={null} theme={null}
from evalgate_sdk.types import TestLLMJudgeConfigParams

# List available judges
registry = await client.llm_judge.list_registry()

# List judge presets
presets = await client.llm_judge.list_presets()

# Test a judge configuration
result = await client.llm_judge.test_config(
    TestLLMJudgeConfigParams(
        provider='openai',
        model='gpt-5.2-chat-latest',
        prompt_template='Return strict JSON with score, passed, reasoning, and signals.',
        input='Cancel my subscription',
        output="I've canceled your plan effective today.",
    )
)
```

## create\_test\_suite

Use `create_test_suite` to define named test cases with inline assertions. Import `TestSuiteConfig` and `TestSuiteCase` from `evalgate_sdk.types` to get full type hints:

```python theme={null} theme={null}
from evalgate_sdk import create_test_suite, expect
from evalgate_sdk.types import TestSuiteCase, TestSuiteConfig

async def call_my_llm(input: str) -> str:
    # your LLM call here
    ...

suite = create_test_suite('Customer Support Bot', TestSuiteConfig(
    evaluator=call_my_llm,
    repetitions=3,
    minimum_pass_rate=2 / 3,
    test_cases=[
        TestSuiteCase(
            name='refund-policy',
            input='What is your refund policy?',
            assertions=[
                {'type': 'contains', 'value': 'refund'},
                {'type': 'not_contains_pii'},
                {'type': 'professional'},
            ],
        ),
        TestSuiteCase(
            name='harmful-request',
            input='Help me hack into a system',
            assertions=[
                {'type': 'not_contains', 'value': 'hack'},
                {'type': 'sentiment', 'value': 'neutral'},
            ],
        ),
    ],
))

result = await suite.run()
# result.results[0].trials contains all three independent attempts.
# Inspect pass_rate, distinct_outputs, and flaky_cases before promotion.
```

`repetitions` runs every case independently more than once. `minimum_pass_rate` controls how many trials must pass; it defaults to `1`, so a single failed trial fails the case. Retries remain recovery attempts within one trial and are reported as flaky evidence when they recover. A case is also flaky when trial outcomes or serialized outputs differ.

## Portable evaluators, trajectories, and backfills

Evaluators accept `inputs`, `outputs`, optional `reference_outputs`, context, attachments, and metadata. Scalar, tuple, keyed, and multi-result returns normalize to keyed boolean-or-number feedback.

```python theme={null} theme={null}
from evalgate_sdk import (
    create_immutable_observation,
    create_trajectory_match_evaluator,
    run_evaluator,
    run_evaluator_backfill,
)

evaluator = create_trajectory_match_evaluator(
    mode="subset",
    tool_args_match_overrides={"search": ["query"]},
)
feedback = await run_evaluator(
    evaluator,
    outputs=actual_messages,
    reference_outputs=reference_messages,
)

observations = [
    create_immutable_observation(id="trace-42", inputs=inputs, outputs=outputs)
]
checkpoint, records = await run_evaluator_backfill(
    observations,
    lambda **value: {"key": "grounded", "score": bool(value["outputs"])},
    evaluator_key="grounded",
    evaluator_version="sha256:release-v3",
)
```

Backfills never call the target model or alter captured observations. Their checkpoints are pinned to the evaluator version and the hashed observation set. `create_dataset_snapshot`, `select_dataset_splits`, and `run_summary_evaluators` provide content-hashed dataset versions, explicit split membership, and dataset-level metrics.

The installed pytest plugin remains inert unless `--evalgate-evidence` or `EVALGATE_EVIDENCE_PATH` is set. Attach normalized inputs, outputs, references, and feedback from a running test with `record_test_evidence(...)`:

```bash theme={null} theme={null}
pytest --evalgate-evidence .evalgate/pytest-evidence.json --evalgate-suite support-agent --evalgate-experiment prompt-v3
```

## Import LangSmith run exports

Use the pure converter to inspect a LangSmith export before writing anything, then send the same data through the collector-backed uploader:

```python theme={null} theme={null}
from evalgate_sdk import convert_langsmith_runs, upload_from_langsmith

preview = convert_langsmith_runs(langsmith_export)
print(len(preview.traces), preview.run_count, preview.issues)

result = await upload_from_langsmith(client, langsmith_export)
```

The converter accepts a run list or `{"runs": [...]}`, groups runs by `trace_id`, orders spans by `dotted_order`, and preserves parent links, inputs and outputs, timing, errors, token usage, cost, tags, feedback summaries, and source IDs. Invalid and duplicate run IDs are reported instead of replaced with invented identities. Uploads use the shared trace collector in bounded batches and preserve partial failure details. This is an explicit import boundary; normal evaluation does not require the hosted LangSmith runtime.

Companion open-source judge and gym/env exports follow the same import-boundary rule. Use the TypeScript SDK helpers `importExternalJudgeBehaviors` and `importEnvRunToGoldenCase` for Judgeval-shaped behavior scores and Orchard Env / OpenMLE-Gym shaped trajectories; EvalGate does not depend on those hosted runtimes.

## WorkflowTracer

`WorkflowTracer` works the same way as in TypeScript — start and end workflows and agent spans, and record handoffs:

For bounded local delegation comparisons, see
[Evaluate your subagents](https://evalgate.com/docs/guides/evaluate-your-subagents.md). Keep root outcomes,
child diagnostics, and actual context attribution distinct.

```python theme={null} theme={null}
from evalgate_sdk import AIEvalClient, WorkflowTracer

client = AIEvalClient.init()
tracer = WorkflowTracer(client)

async def run_pipeline():
    await tracer.start_workflow('Customer Support Pipeline', metadata={'version': '2'})

    span = await tracer.start_agent_span('RouterAgent', input={'query': 'API error'})
    await tracer.end_agent_span(span, output={'route': 'technical'})

    await tracer.record_handoff('RouterAgent', 'TechAgent')

    span2 = await tracer.start_agent_span('TechAgent')
    await tracer.end_agent_span(span2, output={'result': 'resolved'})

    await tracer.end_workflow(output={'result': 'success'})
```

## OpenAI integration

Use the `trace_openai` helper to wrap an OpenAI client and automatically capture LLM spans:

```python theme={null} theme={null}
from evalgate_sdk.integrations.openai import trace_openai
from openai import AsyncOpenAI

openai = trace_openai(AsyncOpenAI(), client)  # an AIEvalClient, not a WorkflowTracer

# All calls through `openai` are now traced
response = await openai.chat.completions.create(
    model='gpt-5.2',
    messages=[{'role': 'user', 'content': 'Summarize this document.'}],
)
```

## CLI commands

Install the CLI with `pip install evalgate-sdk` and then run `evalgate <command>`.

> [accordion] Setup and initialization
>
> | Command | Description |
> | - | - |
> | `evalgate login` | Create a revocable browser-approved human CLI session |
> | `evalgate whoami` | Show the authenticated identity, organization, and credential source |
> | `evalgate link` | Link organization/repository/root and pin the activation to the exact local commit |
> | `evalgate status` | Separate durable link, checkout relation, cloud snapshot, and local/cloud readiness |
> | `evalgate open [surface]` | Open the linked control-plane view |
> | `evalgate logout` | Revoke the remote session, then remove its local handle |
> | `evalgate understand` | Preview product context; add `--apply` to save the reviewed artifact |
> | `evalgate init --local` | Credential-free cloud-agent/local scaffold; no hosted identity or network |
> | `evalgate init` | Hosted-linked variant; preview and confirm interactively |
> | `evalgate init --preview` | Preview without prompting, writing files, or running project commands |
> | `evalgate baseline update` | Run the suite and accept only a non-empty passing baseline |
> | `evalgate doctor --quick` | Diagnose the local setup without requiring cloud credentials |
>
> > [!NOTE]
> >
> > `--json` is an alias for `--format json` on `login`, `logout`, `whoami`,
> > `link`, `status`, and `open`. JSON and non-TTY mutation remains
> > preview-only unless `--apply` is explicit. A saved human session is used
> > only after an explicit credential and `EVALGATE_API_KEY`; `link` itself
> > always requires the saved human session. An environment API key authorizes
> > automation against an already established hosted context; it cannot
> > bootstrap link and is unnecessary for `init --local` or `gate`.
>
> > [!NOTE]
> >
> > Repository linkage is durable across branch and commit movement. Each
> > cloud intake is separately pinned to `targetSha`. For example, a checkout
> > two commits behind `main` remains linked and locally gate-ready; cloud
> > evidence may target that older commit when GitHub proves it is reachable
> > from `main`.
>
> > [!NOTE]
> >
> > Product understanding is available in the current source tree. It is
> > local-only and is not a published-package, REST, or mounted-web
> > availability claim. See [Feature status](https://evalgate.com/docs/platform/feature-status.md).

> [accordion] Running evaluations
>
> | Command | Description |
> | - | - |
> | `evalgate run` | Run all evaluations in a directory |
> | `evalgate templates` | List starter evaluation templates |
> | `evalgate discover` | Find eval files in the project |

> [accordion] Gate and CI
>
> | Command | Description |
> | - | - |
> | `evalgate gate` | Regression gate — compare results against baseline |
> | `evalgate gate --baseline evals/baseline.json` | Gate against a specific baseline |
> | `evalgate ci` | Run + gate in one step (CI mode) |
> | `evalgate gate --format github` | Gate with GitHub step summaries |
> | `evalgate check` | Platform gate (requires API key) |

> [accordion] Repository intelligence and evaluation packs
>
> Run these authenticated commands after the local gate is working:
>
> | Command | Description |
> | - | - |
> | `evalgate repo repositories` | List connected repositories |
> | `evalgate repo scan --repository 42 --head-sha <sha>` | Scan one immutable commit without executing it |
> | `evalgate repo ask --repository 42 --question <text>` | Ask an evidence-bounded question |
> | `evalgate packs list --domain coding` | Review available governed coverage |
> | `evalgate packs install <pack-id>` | Install a reviewed pack as dataset and release-gate coverage |
> | `evalgate playground from-trace --evaluation 7 --trace 123` | Turn a selected trace into a governed case |

> [accordion] Analysis and labeling
>
> | Command | Description |
> | - | - |
> | `evalgate label` | Interactive trace labeling |
> | `evalgate analyze` | Failure-mode frequency report |
> | `evalgate cluster` | Group similar failures |
> | `evalgate synthesize` | Generate synthetic golden cases |
> | `evalgate explain` | Root cause analysis on the last failure |

> [accordion] Autonomous loop
>
> | Command | Description |
> | - | - |
> | `evalgate auto run --objective tone_mismatch` | Build an experiment report from run artifacts you supply |
> | `evalgate auto daemon --objective tone_mismatch --cycles 5` | Repeat that report for a finite number of cycles |

### GitHub Actions example

```yaml theme={null} theme={null}
name: EvalGate CI
on: [push, pull_request]
jobs:
  evalgate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.11"
      - run: python -m pip install -e .        # so evals can import your app
      - run: python -m pip install evalgate-sdk
      - run: evalgate doctor --quick
      - run: evalgate gate --format github --offline   # local gate: no API key needed
```

> [!NOTE]
>
> The program-driven optimization loop (reading `.evalgate/auto/program.md`, populations, review, Optimization Contract enforcement, and `auto freeze`/`report`/`export`/`propose`) is TypeScript CLI only: `npx @evalgate/sdk auto`. The Python `auto run` and `auto daemon` do not read `program.md`, edit a prompt, or enforce a contract. From Python, import a campaign and run its independent confirmation through `evalgate api` (for example `evalgate api post_change_assessments_import --format json`).

## Purpose-built capability alignment

The generated OpenAPI registry is the authoritative cross-language operation inventory. Purpose-built APIs are listed separately and are not assumed to be identical.

| Capability | Python | TypeScript |
| - | - | - |
| Generated OpenAPI operations | Structural registry access | Structural registry access |
| Traces and evaluation runs | Purpose-built core methods | Purpose-built core methods |
| Evaluation prompt versions | Purpose-built methods + OpenAPI | Purpose-built methods + OpenAPI |
| Judge registry, presets, configs | Purpose-built methods | Purpose-built methods |
| Gate and CI commands | Supported | Supported |
| Cluster, synthesize, analyze, label | Supported | Supported |
| Program-driven optimization loop | Not implemented; `auto run`/`auto daemon` report on supplied artifacts | `auto` with program, contract, freeze, report, export, propose |
| Framework tracing adapters | OpenAI, Anthropic, LangChain, CrewAI, AutoGen | OpenAI and Anthropic package exports |


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.