> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Failure patterns

> Named recurring production failure patterns: why they matter, prevalence, related evidence, and the next honest action toward coverage.

# Failure patterns

The product UI titles this surface **Failure patterns** (`/insights`). OpenAPI and storage still use the `failure-topics` resource family; that is an API name, not the operator-facing framing. Canonical public docs host: `evalgate.com/docs` (treat that host as source of truth if a Mintlify preview subdomain lags).

Failure patterns groups observed failure reports into operational cohorts, preserves membership and classifier lineage, and connects topics to Dataset Hub, Playground, Review, regression candidates, and monitors. A resource link or promotion alone does not establish verified protection.

Failure patterns turn authoritative failure reports into named, reviewable cohorts in an observe → understand → cover loop. Each row should answer why the pattern matters, how prevalent it is, and what next action is prepared — not merely inventory status badges. Accepted patterns connect to Dataset Hub, Playground, Review, candidate eval cases, monitors, and active regression coverage.

**Status:** Available (Shipped) · **Owner:** Production Intelligence · **Last verified (historical):** 3.7.0-rc (2026-07-13) · maturity: [Feature status](https://evalgate.com/docs/platform/feature-status.md)

## Open the workspace

Open `/insights`. The workspace has two panes:

* The pattern queue leads with why-it-matters and a next-state cue (for example **Coverage missing**), plus severity, prevalence, and trend when evidence supports them.
* Pattern detail shows impact and deployment segments, membership evidence, **Related evidence** links when correlations exist, definition versions, Model Gateway provenance, human decisions, and downstream corrective resources.

Readers need trace-read permission. Discovery, acceptance, edits, and corrective actions require trace-write permission and at least the member role. Merge and split require the admin role and `admin:workspace` scope. Suppression is a durable audited decision; it does not delete evidence.

## Discover patterns from failure evidence

Select **Discover from failure reports** to analyze open, unsuppressed reports from the previous 30 days. The request is bounded to 1,000 report rows in the UI and 2,000 at the API contract. Narrow the lookback window if the query exceeds that bound.

Discovery requires at least three occurrences by default. A failure report can represent multiple deduplicated occurrences, so its persisted `occurrenceCount` contributes to the evidence threshold without inventing additional membership rows. If no semantic cluster meets the threshold, discovery returns `INSUFFICIENT_EVIDENCE` and leaves the existing workspace unchanged.

The deterministic classifier:

1. orders reports by stable case identity before clustering;
2. normalizes failure descriptions into the canonical failure taxonomy;
3. assigns reports with deterministic embeddings and a configured similarity threshold;
4. fingerprints the normalized failure mode rather than its changing member list; and
5. persists new memberships idempotently.

Rediscovery does not duplicate a topic or rewrite prior membership history. Topic fingerprints remain stable when report order changes or new occurrences reuse the same normalized failure mode.

## Interpret cohorts and impact

Topics can belong to more than one cohort:

| Cohort | Meaning |
| - | - |
| `recurring` | The topic represents at least three occurrences. |
| `novel` | Its first evidence appeared inside the configured novelty window, seven days by default. |
| `severe` | Its highest supported severity is high or critical. |

Impact snapshots distinguish known and unknown data. Report and trace counts, unique affected users/workflows, model, prompt, and deployment segments come from owned trace and failure-report provenance. Cost is summed only from priced Model Gateway ledger records. If none are priced, the UI shows **Unknown**, not `$0`.

Trend values compare persisted snapshots. A merge or split starts a new continuity window and displays **Trend reset**; EvalGate does not present pre-change and post-change membership as a continuous time series. Low-volume history displays **Not enough history** instead of a percentage.

The queue orders severity first, then recurrence and recency. Rare critical failures remain visible ahead of frequent minor failures. Frequency and affected users describe observed evidence. The discovery frame records its lookback, eligible and sampled rows, omissions, and sampling policy; it does not estimate the production population or prove representative coverage.

## Inspect membership evidence

Open **Evidence** to inspect each active membership. A row includes:

* the owned trace key or failure-report ID;
* the canonical category and detector sources;
* assignment source and confidence;
* deduplicated occurrence count;
* a redacted evidence hash; and
* source-membership IDs after a merge or split.

The workspace never sends raw prompts or responses to the label-proposal workflow. Memberships store a hash and explicit redaction state. An organization cannot attach its topic to another organization's report, trace, alert, topic version, action, or membership; database invariants fail closed on those references.

## Review and version a topic

Deterministic discovery creates a proposed topic definition. Review its membership evidence before selecting **Accept proposal**. Acceptance records the actor, expected version, rationale, and before/after state, then activates the exact proposed version.

Human rename, description, severity, status, and owner changes create a new accepted append-only version. They do not edit a historical version. Every decision uses optimistic concurrency. If another operator changes the topic first, the request returns `STALE_PROPOSAL`; refresh and review the new version before deciding.

### Request an AI label proposal

Select **Propose clearer label**, then choose a configured Model Gateway provider and model. EvalGate sends only redacted categories, severity, detector names, confidence, and evidence hashes through the canonical gateway. The resulting name, description, and severity are stored with the gateway `modelCallId` as a **proposal**.

AI-generated labels never activate themselves. A human must inspect the current evidence and accept the proposal. Invalid structured output returns `GATEWAY_OUTPUT_INVALID` and does not create a version.

## Merge, split, suppress, and resolve

### Merge overlapping topics

Select two or more topics, open the intended target, and merge. The request includes the expected version of every source and the target. EvalGate creates a new accepted target version, copies memberships with source-membership lineage, marks old memberships inactive, records decisions on every participating topic, and marks sources as merged. The target starts a new trend continuity window.

### Split a mixed topic

In **Evidence**, select the memberships for the first partition and choose **Split selected**. Every active membership must appear exactly once across at least two non-empty partitions. Unknown, duplicated, or omitted membership IDs return `INVALID_SPLIT` without changing the topic.

Each child receives a stable fingerprint, an accepted split definition, copied membership lineage, and a discontinuous first snapshot. The source remains visible with `split` status for audit history.

### Suppress or resolve

Suppression requires a reason and confirmation. It removes the topic from active operational work while retaining versions, evidence, snapshots, decisions, and downstream links. Resolve a topic when the failure mode is addressed; reopen it by moving it back to active with a new human version and rationale.

## Create corrective coverage

Corrective actions use owned active trace memberships and are idempotent. A repeated idempotency key returns the stored result; reusing the key with different input returns `IDEMPOTENCY_CONFLICT`.

| Action | Result |
| - | - |
| Dataset | Creates a Dataset Hub dataset and adds evidence-backed trace rows. |
| Playground | Creates or reuses trace-derived test cases in the selected evaluation and opens its Playground. |
| Review | If candidate links exist, creates governed `synthetic_candidate` review items; otherwise creates trace review items. |
| Candidates | Creates candidate eval cases in the selected evaluation with trace and failure provenance. |
| Promote approved | Promotes only candidates with an approved adjudication decision into active-gate test cases. Unapproved candidates fail locally with `approved_review_required`. |
| Monitor | Creates a draft Continuous Eval monitor from the topic's canonical failure category. Activate it only after testing its match and budget policy. |

Bulk actions preserve successful items when another item fails. The receipt becomes `partial`, item-level failures remain visible, and completed resource links are retained. A partial response is not reported as complete success.

The governed regression path is:

1. create candidate eval cases from a topic;
2. create candidate review items;
3. record an approved human adjudication decision;
4. promote approved candidates; and
5. inspect the current case, immutable source revision, and independent review;
6. execute the exact case revision with applicable assertion evidence on the relevant gate; and
7. reload the topic to inspect the resulting regression evidence state.

Promotion cannot bypass Review. The topic action checks an approved adjudication source, and the canonical synthetic promotion service independently enforces the candidate lifecycle transition and provenance contract.

Candidate, review, and active-case counts are reconciled from their lifecycle owners on reads. Retired, superseded, or archived cases do not count as active coverage. The detailed evidence map checks reviewed case identity, sealed trace source identity, run-manifest case identity, completed execution, applicable assertion evidence, and the bound evaluation policy. Missing or stale evidence remains explicit. Verified evaluation evidence is post-run evidence; it does not establish that a downstream business action was prevented or that an unobserved production population is covered.

Lists use a continuation cursor. Memberships, versions, snapshots, decisions, links, and regression evidence have bounded pages. Load every active membership page before preparing a split.

## API workflow

| Purpose | Operation |
| - | - |
| Summary | `GET /api/production-insights/summary` |
| List topics | `GET /api/production-insights/topics` with `status`, `severity`, `cohort`, `query`, `limit`, and `cursor` |
| Discover | `POST /api/production-insights/discover` |
| Topic evidence | `GET /api/production-insights/topics/{topicId}` with optional `historySection`, `historyCursor`, `historyLimit`, and `coverageAfterCaseId` |
| Human edit | `PATCH /api/production-insights/topics/{topicId}` |
| Accept proposal | `POST /api/production-insights/topics/{topicId}/accept` |
| Gateway label proposal | `POST /api/production-insights/topics/{topicId}/propose-label` |
| Merge | `POST /api/production-insights/topics/merge` |
| Split | `POST /api/production-insights/topics/{topicId}/split` |
| Corrective action | `POST /api/production-insights/topics/{topicId}/actions/{action}` |

The web workspace uses these shared REST contracts. Generated TypeScript and Python OpenAPI clients and the CLI's `api` command expose operation IDs for the same contracts; there is no separate Failure patterns convenience client.

For example, with an authenticated client, request a bounded history page using `get_production_insights_topics_topicId`:

```typescript theme={null}
import { OpenApiClient } from '@evalgate/sdk/openapi';

const client = new OpenApiClient({ apiKey: process.env.EVALGATE_API_KEY! });
const detail = await client.request('get_production_insights_topics_topicId', {
  path: { topicId },
  query: { historySection: 'versions', historyLimit: 20 },
});
```

```python theme={null}
from evalgate_sdk.openapi_client import OpenApiClient

async with OpenApiClient(api_key=api_key) as client:
    detail = await client.request(
        'get_production_insights_topics_topicId',
        path={'topicId': topic_id},
        query={'historySection': 'versions', 'historyLimit': 20},
    )
```

The CLI uses the same operation contract: `evalgate api get_production_insights_topics --query status=active --query limit=20 --json`. Supply the returned `nextCursor` as `--query cursor=<value>` for the next list page. Detail history uses its section's returned cursor; regression evidence uses `nextCaseId` as `coverageAfterCaseId`.

## Failure and recovery

| Symptom | Meaning | Recovery |
| - | - | - |
| `INSUFFICIENT_EVIDENCE` | Fewer than the required occurrences formed a qualifying cluster, or no owned trace supports an action. | Ingest/analyze more production failures or lower the threshold only when your operating policy allows it. |
| `TOO_MANY_REPORTS` | Discovery exceeded its bounded source-row limit. | Narrow the lookback window; do not cluster a silent partial sample. |
| `STALE_PROPOSAL` | A topic version changed after the operator loaded it. | Refresh, inspect the current evidence/version, and resubmit the decision. |
| `INVALID_MERGE` or `INVALID_SPLIT` | The requested lineage operation is incomplete, duplicated, or version-stale. | Correct the selected topics or membership partitions and retry with current versions. |
| Action returns `partial` | Some trace, candidate, or destination items failed while others completed. | Inspect item errors and retry only failed inputs with a new idempotency key. |
| Promotion says `approved_review_required` | No approved candidate adjudication exists. | Open Review, record an approved decision, then promote again. |
| Trend shows **Trend reset** | A merge or split changed the classifier membership boundary. | Treat the new window as a fresh series; inspect pre-change snapshots in History. |
| Cost shows **Unknown** | No priced Model Gateway ledger record supports the selected evidence. | Repair cost provenance; do not substitute zero. |

## Verification evidence

The release-candidate acceptance evidence covers:

* domain and clustering tests for order stability, fingerprints, cohorts, impact, novelty, continuity, and split validation;
* authenticated route tests for validation, scope/role enforcement, optimistic versions, and action contracts;
* PostgreSQL integration proof for deterministic rediscovery, idempotent membership, and tenant isolation;
* workspace DOM tests for evidence, provenance, empty/insufficient states, and trend discontinuity; and
* authenticated golden path from recurring production failure through accepted topic, candidate, governed review, promotion gate, and active regression coverage.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.