Skip to main content

API rate limits and evaluation usage

EvalGate bills completed evaluation results. API rate limits are separate operational safeguards and do not change the price.

Evaluation usage plans

The hard result-budget and provider-estimate contracts are a source candidate and require the matching deployed release. The prices below do not establish that rollout; review the active limit in Billing. Choose the monthly allowance that fits your evaluation volume: Organizations above 5 million monthly results, rolling EvalGate out across multiple organizations, or requiring procurement and centralized invoicing can use a Strategic Enterprise agreement with a custom annual commitment and negotiated volume schedule. Every plan continues beyond its included allowance at $1 per 1,000 additional results. A hard budget starts at 10,000 results per UTC calendar month, across plans. An administrator must explicitly approve increases. Included allowances still apply; this is a result limit, not a provider-dollar limit. Users, projects, datasets, and runs are unlimited on every plan. Model inference is bring-your-own-key (BYOK). Your model provider bills its own tokens and requests directly; EvalGate does not add an inference markup.

What counts as one result

One result is one persisted test-case outcome from an evaluation run.
  • A passed case counts once.
  • An evaluated failure counts once.
  • Multiple scorers on the same case do not multiply usage.
  • A retry of the same evaluation run reuses the run billing key and does not create a second usage event.
  • An empty run, a human-review placeholder, or an infrastructure failure before a result is persisted counts as zero.
Developer does not require a payment method for its included allowance. A payment method is required to continue beyond that allowance or to subscribe to Team, Scale, or Business. Every tier uses the same $1 per 1,000 overage rate.

API rate limits

Rate limits protect the service from runaway request loops. They are independent of monthly evaluation-result billing. Authenticated requests are keyed by organization and user, or by API key when available. Billing plans can select the corresponding session tier. EvalGate uses Upstash Redis for shared admission. When Redis is unavailable, production sensitive operations fail closed by default; permitted fallback uses bounded state within each process. Anonymous utility requests use Vercel’s platform-provided client address only when running on Vercel. Self-hosted deployments, and requests without a valid trusted address, share one anonymous bucket across utility routes. Caller-sent forwarding headers cannot select a separate bucket. On self-hosted deployments, this means the 100 requests/hour allowance is shared by anonymous visitors; authenticated requests retain their organization, user, or API-key identity. Authenticated requests also consume short-term token buckets: Each bucket replenishes steadily to its listed capacity over 60 seconds. Sensitive operations consume both their sensitive bucket and the shared total bucket. Daily allowances stay unchanged. Successful responses continue to report the daily quota; a burst refusal reports the binding short-term policy and its Retry-After delay. Single and batch collector ingestion share additional organization-wide work buckets: 1,000 traces and 20,000 spans, replenished over 60 seconds. A batch may contain up to 100 traces. Admission uses the actual parsed trace and span counts before any trace is written. A work-limit refusal returns 429, Retry-After, and X-Collector-RateLimit-Policy, -Limit, -Remaining, and -Reset headers. Reduce a batch that exceeds a bucket’s entire capacity before retrying it. Command search admits up to four concurrent searches per process and two per organization within that process, with no waiting queue. Each search dispatches at most two object queries at once. Queries use transaction-local PostgreSQL statement timeouts of at most one second, shortened when less remains in the four-second query-dispatch budget. A timed-out object kind is reported as unavailable; successful kinds retain their ranked results. Full admission returns 429 with Retry-After: 1. These process limits are operational bounds, not a deployment-wide concurrency or availability guarantee.

Rate limit response headers

Every response that reaches rate-limit admission includes quota hints, including successful calls and admitted 400, 401, and 500 responses. Public responses that permit shared caching omit per-caller quota hints. Infrastructure health checks and rejections before admission may also omit them. To inspect a safe, public response, run curl -i https://www.evalgate.com/api/demo/custom-eval. RateLimit-Policy and RateLimit follow the current IETF HTTPAPI structured field draft. Numeric RateLimit-* fields support clients using the earlier HTTPAPI convention; X-RateLimit-* fields retain their existing semantics. Do not interpret RateLimit-Reset as a Unix timestamp. Headers are also listed in Access-Control-Expose-Headers for responses with permitted CORS access. These quota hints are operational controls, not an SLA or a guarantee that a later request will be served. When a request exceeds its limit, the API returns HTTP 429:
Use exponential backoff and respect Retry-After, the t parameter in RateLimit, or X-RateLimit-Reset before retrying. Retry-After takes precedence when it is present.
View current evaluation-result usage or add a payment method from Settings → Billing.