> ## Documentation Index > Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt > Use this file to discover all available pages before exploring further. # Remote Runners > Execute immutable EvalGate release artifacts in customer-controlled infrastructure with signed jobs, bounded capabilities, and complete evidence custody. # Remote Runners Remote Runners let EvalGate dispatch prompts, scorers, tools, workflows, and agent entrypoints into infrastructure that you control. The control plane retains job identity, policy, retry, and evidence authority; the worker receives one signed, immutable execution envelope at a time. Open **Remote Runners** at `/remote-runners` to register workers, inspect capacity and heartbeats, drain or revoke identities, monitor jobs, request cancellation, and review each job's claims, state transitions, stream custody, terminal completion, and reconciliation decisions. > [!NOTE] > > A remote job binds an exact release artifact ID, version ID, SHA-256 content hash, and canonical `dev`, `staging`, or `prod` environment. A runner cannot substitute another version, change the environment, or resolve “latest.” ## Evaluate your real feature with `evalgate serve` `evalgate serve` is the shortest path from a feature in your repository to a graded EvalGate evaluation. It wraps the worker protocol below; nothing about scheduling, grading or evidence custody changes. > [!NOTE] > > `evalgate serve` is experimental. It first shipped in `@evalgate/sdk@3.13.0`. > The worker SDKs below are published. ```bash theme={null} # One time: register this machine as a runner and the feature as a target. export EVALGATE_API_KEY=... # admin key, used only for registration evalgate serve --register \ --target support-answerer --entrypoint src/answer.ts \ --provider openai --model gpt-5-mini --effort low \ --allow-origin https://api.openai.com \ -- node scripts/answer.mjs ``` `--target` registers an **executable target**: the repository commit, root and entrypoint, the declared command, the target model and effort, and the runner requirements, stored as an immutable content-hashed release version. Running the same declaration again reuses that version; a new commit or a different model creates the next one. The registration prints the target's artifact and version IDs, and `GET /api/remote-runners/targets` returns each target's ready-to-use `executorConfig` for an evaluation with `executorType: "remote_runner"`. `serve` declares the target version it registered, and only that version, on every claim. EvalGate hands it only jobs for that exact version and content, so two runners with the same capabilities never run each other's targets; a baseline commit and a candidate commit each need their own `serve`. Without `--target`, `serve` declares that it serves no target and receives no executable-target jobs. A job for another version is refused before the command runs. Each evaluation case runs your command once. The case arrives on stdin: ```json theme={null} {"prompt": "...", "metadata": {}, "evaluationRunId": 42, "testCaseId": 9, "job": {"id": "...", "subject": {...}}, "target": {"provider": "openai", "model": "gpt-5-mini", "effort": "low"}} ``` `EVALGATE_TARGET_MODEL`, `EVALGATE_TARGET_PROVIDER` and `EVALGATE_TARGET_EFFORT` are set when declared. The runner's bootstrap token is removed from the command's environment. Print the answer as plain text, or as JSON with what you observed: ```json theme={null} {"output": "Refunds close after 30 days", "execution": {"mode": "live", "served": {"provider": "openai", "model": "gpt-5-mini-2026-08-01"}, "usage": {"inputTokens": 1200, "outputTokens": 300}, "costUsd": null, "latencyMs": 640, "latencyBoundary": "target_call"}} ``` A non-zero exit fails the case and stderr is streamed as logs. Outliving the job's wall-clock budget stops the command and fails the case. Cancelling the job in EvalGate stops the command at the next heartbeat. > [!WARNING] > > `serve` does not sandbox your command. It runs with your user's permissions > and network access. The declared network origins and limits are recorded on > the target and used to route jobs to compatible runners; they are not > enforced on your machine. ### What EvalGate records for each served case | Field | Source | When missing | | - | - | - | | `mode` | Your command's `execution.mode`, else `--mode` | The whole observation is recorded as `unreported`, never assumed live | | Requested model and effort | `execution.requested`, else `--provider/--model/--effort` | Listed in `missing` | | Served model | `execution.served` as the provider reported it | Listed in `missing` | | Usage | `execution.usage` | Cost is unavailable | | Cost | A billed `costUsd` you report (`runner_reported`), or reported usage priced from EvalGate's exact price table (`evalgate_priced`) | Unavailable, with the reason. EvalGate never prices an unknown model with a guessed rate | | Latency | `execution.latencyMs` with its boundary, else the whole job (`runner_job`) | Not applicable | | Code version | `execution.targetVersion`, else `--target-version`, else git `HEAD` (`-dirty` with local changes) | Listed in `missing` | Both cost provenances are estimates: the tokens and totals come from the runner, not from a gateway EvalGate controls. Simulated runs never record spend. The completion stores this evidence, and the trace EvalGate creates for the execution carries the same record. ### Bind an evaluation from the web On an evaluation that has no target yet, **Connect the system to evaluate** opens on **Code in my repository**. It lists your executable targets with their commit, entrypoint, target model and whether a compatible runner is serving right now, and preselects a served target. For an evaluation drafted from a repository scan it also shows the scanned commit, offers the scan's implementation files as the entrypoint, prefills the `evalgate serve` command, and warns when a target was served from a different commit than the scan. A runner counts as serving while it polls for work or heartbeats a job. To reuse the runner later, export `EVALGATE_RUNNER_ID` and `EVALGATE_RUNNER_BOOTSTRAP_TOKEN` (shown once at registration) and drop `--register`. Use `--once` or `--max-jobs` in CI, and `--format jsonl` for machine-readable progress. ## Permissions | Action | Required scope | | - | - | | List runners and inspect jobs/evidence | `runs:read` | | Enqueue or cancel remote jobs | `runs:write` | | Register, drain, activate, rotate, or revoke a runner | `admin:workspace` and admin role | | Claim, heartbeat, stream, or complete work | Short-lived runner credential with the matching `runner:*` scope | Runner endpoints do not accept a user session or ordinary API key as worker authority. The worker first exchanges its one-time bootstrap secret for a credential that expires within ten minutes and is bound to one organization and runner identity. ## Security and identity model Each registered runner has: * A stable organization-scoped `runnerKey` * A declared protocol version and content-hashed capability document * A maximum concurrent-job count enforced in the database * A one-time bootstrap token stored only as a SHA-256 hash * Short-lived, revocable worker credentials with least-privilege scopes * A derived HMAC-SHA256 job-signing key and visible key ID * Heartbeat, capacity, last-contact, drain, loss, and revocation state Rotating a bootstrap token revokes all outstanding short-lived credentials. Revocation is permanent for that identity; register a replacement runner rather than reactivating a revoked credential boundary. > [!WARNING] > > The bootstrap token is displayed once. Save it directly into your secret manager before closing the dialog. EvalGate cannot recover the plaintext value. ## Register a runner 1. Open `/remote-runners` and choose **Register runner**. 2. Enter a display name and stable runner key. 3. Declare immutable runtime identifiers, such as `nodejs20` or `python3.12`. 4. Select the supported execution kinds. 5. Set the maximum concurrent-job capacity. 6. Review the default-deny isolation boundary and choose **Register identity**. 7. Copy the displayed environment variables and worker start command into your deployment secret and workload configuration. The current registration flow declares network `deny_all`, streaming and cancellation support, one-hour CPU, 64 GiB memory, 24-hour wall time, and a 100 MiB output ceiling. The API accepts a narrower capability document when the worker has smaller limits or a reviewed HTTPS origin allowlist. ## Capability negotiation Before a claim is created, the control plane compares every job requirement with the runner declaration: | Requirement | Enforcement | | - | - | | Execution kind | Must appear in the runner's supported kinds | | Runtime | Exact immutable runtime identifier match | | CPU, memory, wall time, and output | Every requested limit must be at or below the runner ceiling | | Network | `deny_all`, or every required HTTPS origin must appear in the runner allowlist | | Streaming and cancellation | Required support must be explicitly declared | | Concurrency | Active claims must remain below the registered maximum | | Executable target | Only a runner declaring that exact `versionId` and `contentHash` in `servedTargets` receives the job | A runner declares `servedTargets` on registration or in the claim body (`{ "servedTargets": [...] }`), which replaces the registered list. An incompatible runner simply does not receive the job. EvalGate does not silently relax a resource or network requirement to make a claim succeed. ## Start a TypeScript worker The TypeScript SDK verifies the organization binding, key ID, validity window, canonical payload hash, HMAC signature, claim identity, and nonce before returning an assignment to application code. ```ts theme={null} import { RemoteRunnerWorker, createRemoteRunnerCompletion, createRemoteRunnerStreamChunk, } from "@evalgate/sdk"; const worker = new RemoteRunnerWorker({ baseUrl: process.env.EVALGATE_BASE_URL!, runnerId: process.env.EVALGATE_RUNNER_ID!, bootstrapToken: process.env.EVALGATE_RUNNER_BOOTSTRAP_TOKEN!, // The executable-target versions this worker runs. Other targets' jobs are // never offered, and are refused before execution if they arrive. servedTargets: [{ versionId: 42, contentHash: process.env.TARGET_CONTENT_HASH! }], }); const assignment = await worker.claim(); if (!assignment) process.exit(0); const started = createRemoteRunnerStreamChunk({ jobId: assignment.signedJob.envelope.jobId, claimId: assignment.claim.id, dispatchAttempt: assignment.claim.dispatchAttempt, sequence: 0, kind: "log", previousChecksum: null, payload: { level: "info", message: "execution started" }, }); await worker.stream(started); await worker.heartbeat(assignment.claim.id, 60); await worker.complete(createRemoteRunnerCompletion({ jobId: assignment.signedJob.envelope.jobId, claimId: assignment.claim.id, runnerId: process.env.EVALGATE_RUNNER_ID!, organizationId: assignment.signedJob.envelope.organizationId, dispatchAttempt: assignment.claim.dispatchAttempt, status: "succeeded", inputHash: assignment.signedJob.envelope.inputHash, resultHash: resultHash, finalStreamChecksum: started.checksum, evidence: { evaluationRunId, testResultIds: [], // EvalGate resolves the scheduled case costRecordIds: [], modelCallIds: [], traceId: null, // EvalGate records the execution trace }, completedAt: new Date().toISOString(), })); ``` ## Start a Python worker The Python SDK implements the same canonical hashing, signing verification, replay protection, credential refresh, and endpoint contract. ```python theme={null} import os from evalgate_sdk import ( RemoteRunnerWorker, create_remote_runner_completion, create_remote_runner_stream_chunk, ) async with RemoteRunnerWorker( base_url=os.environ["EVALGATE_BASE_URL"], runner_id=os.environ["EVALGATE_RUNNER_ID"], bootstrap_token=os.environ["EVALGATE_RUNNER_BOOTSTRAP_TOKEN"], ) as worker: assignment = await worker.claim() if assignment is None: return chunk = create_remote_runner_stream_chunk( job_id=assignment["signedJob"]["envelope"]["jobId"], claim_id=assignment["claim"]["id"], dispatch_attempt=assignment["claim"]["dispatchAttempt"], sequence=0, kind="trajectory", previous_checksum=None, payload={"event": "agent_started"}, ) await worker.stream(chunk) await worker.heartbeat(assignment["claim"]["id"]) await worker.complete(create_remote_runner_completion( jobId=assignment["signedJob"]["envelope"]["jobId"], claimId=assignment["claim"]["id"], runnerId=os.environ["EVALGATE_RUNNER_ID"], organizationId=assignment["signedJob"]["envelope"]["organizationId"], dispatchAttempt=assignment["claim"]["dispatchAttempt"], status="succeeded", inputHash=assignment["signedJob"]["envelope"]["inputHash"], resultHash=result_hash, finalStreamChecksum=chunk["checksum"], evidence=evidence, completedAt=completed_at, )) ``` ## Enqueue an immutable job Use `POST /api/remote-runners/jobs` with a `runs:write` API key or authenticated session. The request contains: * A stable idempotency key * Subject kind, artifact ID, exact version ID, and `dev`, `staging`, or `prod` environment * A JSON input snapshot * Execution kind, runtime, resource limits, network policy, and streaming/cancellation requirements The server loads the exact release version in the caller's organization, computes and persists its content hash when needed, checks the version's environment, and creates the queued job and audit event atomically. Repeating the same idempotency key with identical intent returns the original job; different intent returns a conflict. ## Signed claim lifecycle ```text theme={null} queued → claimed → running → succeeded | failed │ │ │ └─────────┴──────────┴→ cancel_requested → cancelled │ └→ lost → queued (bounded retry) | cancelled ``` Claim insertion is serialized at the database boundary. A job can have only one active claim and one claim per dispatch attempt. The signed envelope includes the organization, job, attempt, idempotency key, exact subject identity, input hash, requirements, issue/expiry timestamps, and a single-use nonce. The first heartbeat transitions a claimed job to running and extends the lease for 15–120 seconds. It also returns `cancelRequested` and the human cancellation rationale so the worker can stop at a safe boundary. ## Stream logs and trajectories Each `log` or `trajectory` chunk includes: * Job, claim, and dispatch-attempt identity * A zero-based contiguous sequence * The prior chunk checksum, or `null` for sequence zero * A deterministic payload hash * A checksum over identity, sequence, kind, prior checksum, payload hash, and emission time The service and database both reject gaps, reordered chunks, changed claim identity, duplicate sequence numbers, duplicate checksums, and appends to an expired claim. The terminal completion must close the exact last stored checksum. ## Complete with normal EvalGate evidence A completion is accepted only for the active claim and attempt. It binds the input hash, result hash, terminal status, final stream checksum, and normal EvalGate evidence: * Evaluation run ID * Test-result IDs * Cost-record IDs * Model-call ledger IDs * Trace ID A runner usually knows only its claim and the job input. It may send empty test-result, cost and model-call lists and `traceId: null`: EvalGate then resolves the scheduled case's own result rows, records the execution trace, and records the cost from the execution observation on the final trajectory chunk. Identifiers a runner does send are still checked against the organization and the scheduled case. All referenced evidence must belong to the same organization and the exact evaluation run named by the completion; each evidence row may be attached to only one accepted remote completion. Completion insertion, evidence joins, job finalization, claim release, runner-capacity decrement, and the terminal event occur in one transaction. Exact completion replay returns `duplicate`. A different completion after acceptance returns `conflict`. A completion from an earlier dispatch attempt returns `stale`. These decisions are stored with their checksum, rationale, and reported costs rather than discarded. ## Cancellation and runner loss Queued and lost jobs cancel immediately. Claimed and running jobs enter `cancel_requested`; the worker sees that state on heartbeat. If cancellation wins the terminal race, the job becomes cancelled while already-incurred provider or sandbox costs remain authoritative. When a heartbeat lease expires, the reaper: 1. Marks the claim and runner lost. 2. Releases the runner's active capacity. 3. Appends a durable loss event. 4. Requeues the job with no active claim when fewer than three attempts have been dispatched. 5. Uses a new nonce and incremented attempt for the next compatible runner. EvalGate invokes bounded recovery from the existing `CRON_SECRET`-authenticated job runner. Organization admins can also trigger the same bounded recovery from the control plane during incident response; both paths retain the durable loss and requeue events. An older worker cannot complete the replacement attempt. Accepted results, test results, model calls, and cost records have unique custody constraints, preventing duplicate results or charges. ## Operate and troubleshoot | Symptom | What to check | | - | - | | Session returns `401` | Runner ID and current bootstrap token; rotate if the token may be exposed | | Session returns `503` | `REMOTE_RUNNER_SIGNING_SECRET` must be configured with at least 32 characters | | Claim returns `204` | No compatible queued job, or another runner claimed the available work | | Claim returns `503` | Runner is draining, lost, revoked, or at concurrent capacity | | Job stays claimed | Worker must heartbeat; the first heartbeat records execution start | | Stream returns `422` or `409` | Verify sequence, prior checksum, claim/attempt identity, and active lease | | Completion is `stale` | A newer dispatch attempt replaced this runner after lease loss | | Completion is `conflict` | Another terminal checksum was already accepted for this job | Use the **Jobs** tab and **Evidence** action to inspect the complete state timeline and reconciliation history before retrying or rotating a worker. This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.