> ## Documentation Index
> Fetch the complete documentation index at: https://evalgate.com/docs/llms.txt
> Use this file to discover all available pages before exploring further.

# Annotations

# Annotations API — human labeling and review

> Create annotation tasks, add items for human review, and submit labels to build the golden dataset used for measuring LLM judge credibility.

Human annotations are the foundation of judge credibility in EvalGate. When your team labels outputs, those labels become the ground truth that automated judge scores are compared against. Annotation tasks organize labeling work; run-case ratings (`POST /api/annotations`, below) feed the [LLM Judge alignment endpoint](https://evalgate.com/docs/api/llm-judge.md). A judge with high alignment against a well-labeled dataset is one you can trust to gate your CI pipeline.

## GET /api/annotations/tasks — list annotation tasks

Returns annotation tasks for the authenticated organization, most recently updated first. Filter with `status` or `search` (name), and paginate with `limit` (default 50) and `offset`. Pass `id` to fetch a single task.

```bash theme={null} theme={null}
curl "https://evalgate.com/api/annotations/tasks?limit=20" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Response

```json theme={null} theme={null}
{
  "tasks": [
    {
      "id": 12,
      "name": "Support quality review",
      "description": null,
      "type": "binary",
      "status": "active",
      "totalItems": 120,
      "completedItems": 87,
      "organizationId": "00000000-0000-4000-8000-000000000001",
      "createdAt": "2026-09-01T09:00:00.000Z",
      "updatedAt": "2026-09-10T14:22:00.000Z"
    }
  ],
  "hasMore": false,
  "total": 1,
  "limit": 20,
  "offset": 0
}
```

**tasks** `array`

**Task fields**

**id** `integer`

Unique task ID.

**name** `string`

Display name for the task.

**type** `string`

Task type, such as `binary`, `rating`, or `checklist`.

**status** `string`

Task status. New tasks start as `draft`.

**totalItems** `integer`

Total number of items in this task.

**completedItems** `integer`

Number of items that have been completed.

**annotationSettings** `object`

Instructions, label fields, and reviewer assignment settings.

**organizationId** `string`

Owning organization UUID.

**createdAt** `string`

ISO 8601 creation timestamp.

***

## POST /api/annotations/tasks — create an annotation task

Creates a new task and defines the label fields annotators fill in. Add the items to review with `POST /api/annotations/tasks/{id}/items`.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations/tasks \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "Support quality review",
    "type": "binary",
    "instructions": "Mark each response pass or fail.",
    "fields": [
      { "id": "verdict", "name": "verdict", "label": "Verdict", "type": "binary", "options": ["pass", "fail"], "required": true }
    ]
  }'
```

### Request body

**name** `string` *required*

Display name for this annotation task.

**type** `string` *required*

Task type, such as `binary`, `rating`, or `checklist`.

**description** `string`

Optional description of the task.

**instructions** `string`

Optional guidance shown to annotators when they open the task.

**fields** `array`

Up to 40 label fields. Each field has a `name`, `label`, and `type` (`binary`, `categorical`, `rating`, `multi_select`, `text`, or `checklist`), plus optional `id`, `options`, `scale`, and `required`. A field's `id` defaults to its `name`.

### Response (201)

Returns the created task. New tasks start in `draft` status with `totalItems: 0`.

***

## GET /api/annotations/tasks/{id} — get a task

Returns a single annotation task as `{ "task": { ... } }`.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations/tasks/12 \
  -H "Authorization: Bearer YOUR_API_KEY"
```

***

## GET /api/annotations/tasks/{id}/items — list task items

Returns the items in a task with a task summary. Pass `annotated=true` or `annotated=false` to filter, and paginate with `limit` and `offset`.

```bash theme={null} theme={null}
curl "https://evalgate.com/api/annotations/tasks/12/items?annotated=false" \
  -H "Authorization: Bearer YOUR_API_KEY"
```

### Response

```json theme={null} theme={null}
{
  "items": [
    {
      "id": 202,
      "taskId": 12,
      "content": "Q: Cancel my subscription\nA: I've canceled your plan effective today.",
      "subjectKind": "text",
      "annotation": null,
      "annotatedBy": null,
      "annotatedAt": null,
      "status": "pending",
      "createdAt": "2026-09-01T09:05:00.000Z"
    }
  ],
  "task": { "id": 12, "name": "Support quality review", "type": "binary", "status": "active", "totalItems": 120, "completedItems": 87 },
  "hasMore": false,
  "limit": 50,
  "offset": 0
}
```

***

## POST /api/annotations/tasks/{id}/items — add an item

Adds one item to review. Returns `201` with the created item.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations/tasks/12/items \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "content": "Q: Cancel my subscription\nA: I have canceled your plan effective today.",
    "subjectKind": "trace_span",
    "subjectRef": { "traceId": 42 }
  }'
```

**content** `string` *required*

The text the annotator reviews.

**subjectKind** `string`

What the item refers to: `text` (default), `evaluation_result`, `trace_span`, `dataset_row`, or `evidence_item`.

**subjectRef** `object`

Identifiers for the referenced subject, such as a trace or evaluation result ID.

***

## PUT /api/annotations/tasks/{id}/items?itemId={itemId} — submit an annotation

Submits answers for one item. The annotator is always the authenticated caller.

```bash theme={null} theme={null}
curl "https://evalgate.com/api/annotations/tasks/12/items?itemId=202" \
  -X PUT \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "answers": [{ "fieldId": "verdict", "value": "fail" }],
    "feedback": "Response did not address the user'\''s core question"
  }'
```

### Request body

**answers** `array`

One entry per field: `fieldId`, `value` (string, number, boolean, string array, or `null`), and optional `unableToJudge`, `notApplicable`, and `rationale`.

**feedback** `string`

Optional free-text notes explaining the label decision, stored for audit and inter-rater review.

**draft** `boolean`

Save without completing required-field validation.

**defer** `boolean`

Defer the item without completing it.

**expectedWriteRevision** `integer`

Optimistic-concurrency guard. A stale revision returns `409 CONFLICT`.

### Response

```json theme={null} theme={null}
{
  "item": { "id": 202, "taskId": 12 },
  "completed": true,
  "deferred": false,
  "revision": 1,
  "writeRevision": 1,
  "pendingOtherReviewers": false,
  "reviewerCount": 1,
  "requiredReviewers": 1
}
```

***

## POST /api/annotations — rate an evaluation run case

Records a human rating for one test case in an evaluation run. These ratings are what `GET /api/llm-judge/alignment` compares with judge scores. `GET /api/annotations?evaluationRunId=` lists them.

```bash theme={null} theme={null}
curl https://evalgate.com/api/annotations \
  -X POST \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "evaluationRunId": 42,
    "testCaseId": 301,
    "rating": 5,
    "feedback": "Correct and complete."
  }'
```

**evaluationRunId** `integer` *required*

The evaluation run that produced the output.

**testCaseId** `integer` *required*

The test case being rated.

**rating** `integer`

Rating from 1 to 5.

**feedback** `string`

Optional free-text notes.

**taskId** `integer`

Optional annotation task this rating belongs to.

> [!NOTE]
>
> Once a task has enough labels, run the [LLM Judge alignment check](https://evalgate.com/docs/api/llm-judge.md) to compare your automated judge with your team's ground truth, and measure TPR and TNR before using the judge as an automated CI gate.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.