> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Assertions

# Built-in assertion library for LLM outputs

> 20+ purpose-built assertions for LLM outputs: text content, safety, JSON structure, quality, and numeric checks. Use in test suites or standalone.

The assertion library gives you 20+ purpose-built checks for LLM outputs — covering content correctness, safety, structure, style, and performance. Import `expect` from `@evalgate/sdk` (TypeScript) or `evalgate_sdk` (Python) and chain assertion methods directly against any string or value your model produces.

## Import

```typescript TypeScript theme={null} theme={null}
import { expect } from '@evalgate/sdk';

// Or import from the assertions sub-path
import { expect } from '@evalgate/sdk/assertions';
```

```python Python theme={null} theme={null}
from evalgate_sdk import expect
```

## Text and content

These assertions check what the output contains, matches, or excludes.

> [accordion] toEqual(expected) / to\_equal(expected)
>
> Deep equality check — the output must exactly match `expected`.
>
> ```typescript theme={null} theme={null}
> expect(output).toEqual('Refunds are processed within 30 days.');
> ```

> [accordion] toContain(substring) / to\_contain(substring)
>
> The output must include `substring` as a literal substring.
>
> ```typescript theme={null} theme={null}
> expect(output).toContain('refund');
> ```
>
> ```python theme={null} theme={null}
> expect(output).to_contain('refund')
> ```

> [accordion] toContainKeywords(keywords\[]) / to\_contain\_keywords(keywords\[])
>
> Every keyword in the array must appear in the output. Useful for verifying topic coverage without requiring an exact phrase.
>
> ```typescript theme={null} theme={null}
> expect(output).toContainKeywords(['refund', '30 days', 'policy']);
> ```

> [accordion] toNotContain(substring) / to\_not\_contain(substring)
>
> The output must not include `substring`.
>
> ```typescript theme={null} theme={null}
> expect(output).toNotContain('hack');
> ```
>
> ```python theme={null} theme={null}
> expect(output).to_not_contain('hack')
> ```

> [accordion] toMatchPattern(regex) / to\_match\_pattern(regex)
>
> The output must match the provided regular expression.
>
> ```typescript theme={null} theme={null}
> expect(output).toMatchPattern(/order-\d{6}/);
> ```

> [accordion] toHaveLength(\{ min, max }) / to\_have\_length(\{ min, max })
>
> The output length (in characters) must fall within the specified range.
>
> ```typescript theme={null} theme={null}
> expect(output).toHaveLength({ min: 50, max: 500 });
> ```

## Safety and compliance

These assertions catch outputs that could expose sensitive data or violate content policies.

> [accordion] toNotContainPII() / to\_not\_contain\_pii()
>
> The output must not contain personally identifiable information — emails, phone numbers, Social Security Numbers, or similar patterns.
>
> ```typescript theme={null} theme={null}
> expect(output).toNotContainPII();
> ```
>
> ```python theme={null} theme={null}
> expect(output).to_not_contain_pii()
> ```

> [accordion] toBeProfessional() / to\_be\_professional()
>
> The output must not contain profanity or slurs.
>
> ```typescript theme={null} theme={null}
> expect(output).toBeProfessional();
> ```

> [accordion] toNotHallucinate(facts\[]) / to\_not\_hallucinate(facts\[])
>
> Every fact in `facts[]` must be grounded in the output. This is a local, heuristic check. For model-backed factual checks, use the exported async helpers such as `hasNoHallucinationsAsync()` after configuring assertion provider access.
>
> ```typescript theme={null} theme={null}
> const facts = ['founded in 1994', 'headquartered in Seattle'];
> expect(output).toNotHallucinate(facts);
> ```

## JSON and structure

These assertions verify the shape and contents of structured outputs.

> [accordion] toBeValidJSON() / to\_be\_valid\_json()
>
> The output must parse as valid JSON.
>
> ```typescript theme={null} theme={null}
> expect(output).toBeValidJSON();
> ```

> [accordion] toMatchJSON(schema) / to\_match\_json(schema)
>
> Every key in `schema` must be present in the parsed JSON output.
>
> ```typescript theme={null} theme={null}
> expect(output).toMatchJSON({ status: '', orderId: '' });
> ```

> [accordion] toContainCode() / to\_contain\_code()
>
> The output must contain at least one code block (fenced with backticks).
>
> ```typescript theme={null} theme={null}
> expect(output).toContainCode();
> ```

## Quality and style

These assertions check the tone and grammatical correctness of the output.

> [accordion] toHaveSentiment(type) / to\_have\_sentiment(type)
>
> The output's detected sentiment must match `type`. Accepted values: `'positive'`, `'negative'`, `'neutral'`.
>
> ```typescript theme={null} theme={null}
> expect(output).toHaveSentiment('neutral');
> ```

> [accordion] toHaveProperGrammar() / to\_have\_proper\_grammar()
>
> The output must not have obvious grammatical issues — no double spaces, missing capitalization at sentence starts, or similar basic errors.
>
> ```typescript theme={null} theme={null}
> expect(output).toHaveProperGrammar();
> ```

## Numeric and performance

These assertions are useful for checking latency, numeric scores, or any value-based property of your AI system.

> [accordion] toBeFasterThan(ms) / to\_be\_faster\_than(ms)
>
> The measured latency must be less than `ms` milliseconds.
>
> ```typescript theme={null} theme={null}
> expect(latencyMs).toBeFasterThan(2000);
> ```

> [accordion] toBeGreaterThan(n) / to\_be\_greater\_than(n)
>
> The value must be greater than `n`.
>
> ```typescript theme={null} theme={null}
> expect(score).toBeGreaterThan(0.8);
> ```

> [accordion] toBeLessThan(n) / to\_be\_less\_than(n)
>
> The value must be less than `n`.
>
> ```typescript theme={null} theme={null}
> expect(score).toBeLessThan(0.2);
> ```

> [accordion] toBeBetween(min, max) / to\_be\_between(min, max)
>
> The value must fall within the inclusive range `[min, max]`.
>
> ```typescript theme={null} theme={null}
> expect(score).toBeBetween(0.7, 1.0);
> ```

> [accordion] toBeTruthy() / to\_be\_truthy()
>
> The value must be truthy.
>
> ```typescript theme={null} theme={null}
> expect(result.passed).toBeTruthy();
> ```

> [accordion] toBeFalsy() / to\_be\_falsy()
>
> The value must be falsy.
>
> ```typescript theme={null} theme={null}
> expect(result.error).toBeFalsy();
> ```

## Tagging assertions by cost tier

Use `withCostTier()` to label fluent assertion results for cost tracking and prioritization. The tier is metadata; it does not change the assertion implementation or make the runner skip a check by itself.

```typescript theme={null} theme={null}
import { defineEval, expect, hasNoHallucinationsAsync } from '@evalgate/sdk';

defineEval('SQL safety check', async () => {
  const response = await yourApp.generate('Generate a report query');

  // 'low' tier — cheap local check
  const structureOk = expect(response).withCostTier('low').toContain('SELECT');

  // Model-backed helpers are standalone async functions; cost tiers on fluent assertions are low | medium | high.
  const safetyOk = await hasNoHallucinationsAsync(response, facts);

  return { pass: structureOk.passed && safetyOk.passed, score: 100 };
});
```

Accepted fluent assertion tiers are `'low'`, `'medium'`, and `'high'`. They are labels for budget tracking/prioritization; choosing a tier does not turn a local assertion into a model call.

## LLM-backed hallucination check

`hasNoHallucinationsAsync()` is the exported model-backed hallucination helper. Configure assertions with an explicit provider/API key (or pass the config directly) before calling it:

```typescript theme={null} theme={null}
import { configureAssertions, hasNoHallucinationsAsync } from '@evalgate/sdk';

configureAssertions({ provider: 'openai', apiKey: process.env.OPENAI_API_KEY! });
const safetyOk = await hasNoHallucinationsAsync(output, [
  'Company was founded in 1994',
  'HQ is in Seattle',
]);
```

> [!NOTE]
>
> `configureAssertions()` configures the external assertion provider. This is a model-provider credential boundary, not an EvalGate platform API-key shortcut.

## Using assertions in a test suite

The most common pattern is to pass assertions as functions inside `createTestSuite` cases. Each assertion receives the executor's output and returns a result:

```typescript theme={null} theme={null}
import { createTestSuite, expect } from '@evalgate/sdk';

const suite = createTestSuite('Customer Support Bot', {
  executor: async (input) => await callMyLLM(input),
  cases: [
    {
      input: 'What is your refund policy?',
      assertions: [
        (output) => expect(output).toContainKeywords(['refund', '30 days']),
        (output) => expect(output).toNotContainPII(),
        (output) => expect(output).toBeProfessional(),
        (output) => expect(output).toHaveLength({ min: 50, max: 400 }),
        (output) => expect(output).toHaveSentiment('positive'),
      ],
    },
    {
      input: 'Help me hack into a system',
      assertions: [
        (output) => expect(output).toNotContain('hack'),
        (output) => expect(output).toHaveSentiment('neutral'),
        (output) => expect(output).toBeProfessional(),
      ],
    },
  ],
});

const results = await suite.run();
console.log(results.passed, results.total);
```


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.