Annotations API — human labeling and review
Create annotation tasks, add items for human review, and submit labels to build the golden dataset used for measuring LLM judge credibility.Human annotations are the foundation of judge credibility in EvalGate. When your team labels outputs, those labels become the ground truth that automated judge scores are compared against. Annotation tasks organize labeling work; run-case ratings (
POST /api/annotations, below) feed the LLM Judge alignment endpoint. A judge with high alignment against a well-labeled dataset is one you can trust to gate your CI pipeline.
GET /api/annotations/tasks — list annotation tasks
Returns annotation tasks for the authenticated organization, most recently updated first. Filter withstatus or search (name), and paginate with limit (default 50) and offset. Pass id to fetch a single task.
Response
array
POST /api/annotations/tasks — create an annotation task
Creates a new task and defines the label fields annotators fill in. Add the items to review withPOST /api/annotations/tasks/{id}/items.
Request body
string
required
Display name for this annotation task.
string
required
Task type, such as
binary, rating, or checklist.string
Optional description of the task.
string
Optional guidance shown to annotators when they open the task.
array
Up to 40 label fields. Each field has a
name, label, and type (binary, categorical, rating, multi_select, text, or checklist), plus optional id, options, scale, and required. A field’s id defaults to its name.Response (201)
Returns the created task. New tasks start indraft status with totalItems: 0.
GET /api/annotations/tasks/ — get a task
Returns a single annotation task as{ "task": { ... } }.
GET /api/annotations/tasks//items — list task items
Returns the items in a task with a task summary. Passannotated=true or annotated=false to filter, and paginate with limit and offset.
Response
POST /api/annotations/tasks//items — add an item
Adds one item to review. Returns201 with the created item.
string
required
The text the annotator reviews.
string
What the item refers to:
text (default), evaluation_result, trace_span, dataset_row, or evidence_item.object
Identifiers for the referenced subject, such as a trace or evaluation result ID.
PUT /api/annotations/tasks//items?itemId= — submit an annotation
Submits answers for one item. The annotator is always the authenticated caller.Request body
array
One entry per field:
fieldId, value (string, number, boolean, string array, or null), and optional unableToJudge, notApplicable, and rationale.string
Optional free-text notes explaining the label decision, stored for audit and inter-rater review.
boolean
Save without completing required-field validation.
boolean
Defer the item without completing it.
integer
Optimistic-concurrency guard. A stale revision returns
409 CONFLICT.Response
POST /api/annotations — rate an evaluation run case
Records a human rating for one test case in an evaluation run. These ratings are whatGET /api/llm-judge/alignment compares with judge scores. GET /api/annotations?evaluationRunId= lists them.
integer
required
The evaluation run that produced the output.
integer
required
The test case being rated.
integer
Rating from 1 to 5.
string
Optional free-text notes.
integer
Optional annotation task this rating belongs to.
Once a task has enough labels, run the LLM Judge alignment check to compare your automated judge with your team’s ground truth, and measure TPR and TNR before using the judge as an automated CI gate.