> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Rag evaluation

# Evaluate RAG systems for accuracy and grounding

> Measure retrieval quality, context relevance, and answer faithfulness in RAG pipelines to catch hallucinations and retrieval failures before users do.

Retrieval-Augmented Generation systems fail in ways that are hard to detect without structured evaluation. A model might retrieve the wrong documents, retrieve the right documents but ignore them, or generate plausible-sounding answers that contradict the retrieved context. Because each component can fail independently, you need to evaluate retrieval, context relevance, faithfulness, and answer quality both separately and end-to-end. This guide shows you how to build that evaluation framework with EvalGate.

## Why RAG evaluation is hard

RAG systems have four distinct failure points, and passing on one doesn't mean passing on the others:

| Failure point | Question to answer |
| - | - |
| **Retrieval** | Did you find the right documents? |
| **Relevance** | Is the retrieved context actually useful for the query? |
| **Generation** | Did the LLM use the context correctly? |
| **Grounding** | Is the answer supported by the retrieved docs, or is it hallucinating? |

Evaluating only the final output hides which component broke. Evaluate each layer separately to diagnose and fix issues efficiently.

## Retrieval quality metrics

Before evaluating end-to-end, isolate the retrieval step. Given a query and a set of known-relevant documents (your gold standard), measure:

| Metric | What it measures |
| - | - |
| **Precision\@K** | Of the top K retrieved docs, what fraction are relevant? |
| **Recall\@K** | Of all relevant docs, what fraction appear in the top K? |
| **MRR (Mean Reciprocal Rank)** | Mean of `1/rank` of the first relevant document per query (1 if relevant at rank 1, 0.5 at rank 2, 0 if none) |
| **NDCG** | Quality of the full ranking, weighted by position |

> [!NOTE]
>
> Missing relevance labels are **unknown**, not irrelevant. Do not treat unlabeled documents as negatives when computing precision, recall, or MRR. For EvalGate's reference-labelled lexical workflow and metric boundaries, see [Measure retrieval quality](https://evalgate.com/docs/guides/measure-retrieval-quality.md). Sample helper snippets in this guide are conceptual; use documented SDK methods and that workflow for supported retrieval measurement.

Example calculation:

```
Query:      "What is our refund policy for damaged items?"
Gold docs:  [doc_42, doc_87, doc_103]
Retrieved:  [doc_42, doc_91, doc_103, doc_45, doc_87]

Precision@3 = 2/3 = 67%    (doc_42 and doc_103 are relevant)
Recall@5    = 3/3 = 100%   (all gold docs found in top 5)
```

Test retrieval in isolation before running the full pipeline:

```typescript TypeScript theme={null} theme={null}
// Test just the retrieval step
const retrieved = await vectorDB.search(query, { k: 5 })

const relevantDocs = ['docs/upload-limits.md', 'docs/faq.md']
const precision = retrieved.filter(doc =>
  relevantDocs.includes(doc.id)
).length / 5

console.log(`Precision@5: ${precision * 100}%`)
```

## Context relevance scoring

Even when the right documents are retrieved, the retrieved chunks may not be useful for the specific query. Use an LLM judge to score relevance on a 1–5 scale:

```
Given this query: [QUERY]
And retrieved context: [CONTEXT]

Rate how relevant this context is for answering the query on a scale of 1–5.
5 = the context directly answers the query.
1 = the context is completely irrelevant.
```

Track **context precision** (the percentage of retrieved chunks that are actually relevant) alongside the raw relevance score to catch cases where your retriever is padding results with noise.

## Answer faithfulness and hallucination detection

Faithfulness measures whether every claim in the generated answer is supported by the retrieved context. EvalGate's `expect(answer).toNotHallucinate(facts)` matcher is only a lightweight check that required ground-truth facts appear in the answer; for claim-level faithfulness, verify each claim against the context:

```typescript TypeScript theme={null} theme={null}
// Extract claims from the answer, then verify each against context
const claims = await extractClaims(answer)

const supported = await Promise.all(
  claims.map(claim => isSupported(claim, context))
)

const faithfulness = supported.filter(Boolean).length / claims.length

// Assert 95%+ of claims must be grounded in retrieved context
assert(faithfulness >= 0.95)
```

Common hallucination pattern to test for:

```
Context:    "Our support team responds within 24 hours on weekdays."
Bad answer: "Support responds within 24 hours, including weekends."
```

The LLM added information that wasn't in the context. Catch this by asserting every factual claim has a supporting source.

## End-to-end RAG evaluation

Combine retrieval checks, faithfulness, and answer correctness into a single end-to-end test:

**1. Create query-answer pairs**

Collect 100–200 representative queries with gold-standard answers and the document IDs that should be retrieved:

```typescript TypeScript theme={null} theme={null}
const testCase = {
  query: "What is the maximum file size for uploads?",
  expectedAnswer: "The maximum file size is 100MB per file",
  relevantDocs: ["docs/upload-limits.md", "docs/faq.md"],
  category: "Technical specs"
}
```

**2. Test retrieval separately**

Run the retrieval step in isolation first. Fix retrieval failures before you debug generation — generation quality is meaningless if the context is wrong.

```typescript TypeScript theme={null} theme={null}
const retrieved = await vectorDB.search(testCase.query, { k: 5 })

const relevantDocs = testCase.relevantDocs
const precision = retrieved.filter(doc =>
  relevantDocs.includes(doc.id)
).length / 5

console.log(`Precision@5: ${precision * 100}%`)
```

**3. Evaluate end-to-end**

Run the full pipeline and assert on retrieval quality, answer correctness, and faithfulness:

```typescript TypeScript theme={null} theme={null}
const result = await ragPipeline.query("What is the max file size?")

// Check retrieval quality
assert(result.retrievedDocs.some(doc => doc.id === "docs/upload-limits.md"))

// Check answer correctness (LLM judge, scored 1–5)
const correctness = await llmJudge.evaluate(result.answer, expectedAnswer)
assert(correctness >= 4)

// Check faithfulness (% of claims grounded in context)
const faithfulness = await checkFaithfulness(result.answer, result.context)
assert(faithfulness >= 0.95)
```

## Common failure modes

> [accordion] Retrieval failures: terminology mismatch
>
> **Problem:** The query uses different words than the documents.
>
> * Query: "How do I reset my password?"
> * Documents use: "password recovery" not "reset"
>
> **Solution:** Add query expansion, synonym handling, or hybrid search (keyword + semantic).

> [accordion] Context window overflow
>
> You retrieved too many documents and exceeded the LLM's context limit, causing truncation of relevant content.
>
> **Solution:** Rerank retrieved chunks and truncate to the most relevant subset before passing to the model.

> [accordion] Incomplete answers
>
> Relevant information was retrieved but not included in the final answer.
>
> **Solution:** Improve the generation prompt to explicitly instruct the model to address all relevant points from the provided context.

## Advanced retrieval techniques

### Hybrid search

Combine semantic and keyword search, then merge rankings:

```typescript TypeScript theme={null} theme={null}
const semanticResults = await vectorDB.search(embedding, { k: 10 })
const keywordResults = await fullTextSearch(query, { k: 10 })

// Merge using reciprocal rank fusion
const combined = reciprocalRankFusion(semanticResults, keywordResults)
```

### Query rewriting

Rewrite ambiguous user queries into clearer search queries before hitting the retriever:

```
Original:  "How do I do that thing with the files?"
Rewritten: "How to upload files? What is the file size limit?"
```

### Multi-hop retrieval

For complex queries that require information from multiple sources:

1. Retrieve documents answering the first part of the query
2. Use those results to refine the query for a second retrieval pass
3. Combine information from both retrievals before generating

## Production metrics to track continuously

Monitor these signals in the EvalGate dashboard after deployment:

* **Answer rate** — percentage of queries answered vs. "I don't know" responses
* **User feedback** — thumbs up/down signals from users
* **Retrieval latency** — time to fetch and rank documents
* **Generation latency** — time to produce the final answer
* **Context usage** — whether retrieved docs are actually referenced in answers

> [!TIP]
>
> Convert production traces into regression test cases. When a user reports a bad answer, capture that query and the expected answer as a test case to prevent the same failure from recurring.

## Related guides

* [Measure retrieval quality](https://evalgate.com/docs/guides/measure-retrieval-quality.md) — Reference-labelled lexical workflow, missing-reference handling, and metric boundaries
* [Compare evaluators](https://evalgate.com/docs/guides/compare-evaluators.md) — Keep application responses fixed when measuring checkers
* [Read evaluation evidence](https://evalgate.com/docs/guides/read-evaluation-evidence.md) — Observation versus terminal outcome


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.