API test mode

Use the public assertion API to test an HTTP integration before connecting a workspace. It runs the same deterministic SDK assertions used by the public playground against the text you supply. No API key or account is required. This is a stateless test mode: it cannot create, read, update, or delete customer evaluations, traces, datasets, provider keys, or billing records. It does not invoke a model provider or generate billable evaluation results. Normal access and API-usage metadata and rate-limit counters still apply. Send synthetic examples rather than confidential data.

Discover and run

The base URL is https://www.evalgate.com. Discover the test-mode contract:

curl -i https://www.evalgate.com/api/demo/custom-eval

The JSON descriptor lists supported assertions, limits, and an executable example. Submit that example to the same endpoint:

curl -i https://www.evalgate.com/api/demo/custom-eval \
  -H 'Content-Type: application/json' \
  --data '{"output":"{\"answer\":42}","assertions":["valid-json"]}'

A successful request returns HTTP 200, mode: "test", persisted: false, providerCalls: false, billable: false, and assertion results. The example passes one assertion. Change output to not json to observe a failed assertion: HTTP 200 still means the request executed, while results.failed becomes 1. Invalid JSON or invalid input returns 400; a body above 64 KiB returns 413. Bodies are limited to 65,536 bytes, with at most 32 assertions and 32 keywords per request. Results exist only in the response, so there is no saved test state to reset.

Generated clients and CLI

The OpenAPI operations are getAssertionTestMode and runAssertionTestMode. Use a client release containing those IDs; if your installed client predates them, use the HTTP examples above. Credential-free access applies only to operations whose OpenAPI contract declares public access.

import { OpenApiClient } from "@evalgate/sdk/openapi";

const api = new OpenApiClient();
const mode = await api.request("getAssertionTestMode");
const result = await api.request("runAssertionTestMode", {
  body: { output: '{"answer":42}', assertions: ["valid-json"] },
});
console.log(mode.mode, result.results.passed);

import asyncio
from evalgate_sdk.openapi_client import OpenApiClient

async def main():
    async with OpenApiClient() as api:
        result = await api.request(
            "runAssertionTestMode",
            body={"output": '{"answer":42}', "assertions": ["valid-json"]},
        )
        print(result["results"]["passed"])

asyncio.run(main())

evalgate api getAssertionTestMode --json
evalgate api runAssertionTestMode \
  --body '{"output":"{\"answer\":42}","assertions":["valid-json"]}' --json

Quotas and scope

Test-mode requests share the anonymous allowance of 100 requests/hour. Vercel uses the trusted platform client IP; self-hosted or untrusted ingress shares one anonymous bucket. Both successful and admitted error responses carry the current RateLimit/RateLimit-Policy, numeric RateLimit-*, and legacy X-RateLimit-* headers. On 429, wait at least Retry-After seconds. See rate-limit conventions. This test mode covers deterministic assertions and their HTTP contract. It does not reproduce hosted CRUD, tenant lifecycle, model behavior, or payment settlement. Those operations require an explicitly disposable environment; setting a test header or using a free production workspace does not isolate them.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Discover and run

Generated clients and CLI

Quotas and scope

​Discover and run

​Generated clients and CLI

​Quotas and scope

Discover and run

Generated clients and CLI

Quotas and scope