Traces: visibility into LLM behavior

Learn what traces and spans are, what data they capture, how sampling works, and how to promote real failures into evaluation test cases.

A trace is a structured record of one complete run of your AI system — everything that happened from the moment a request arrived to the moment a response was returned. Traces give you the ground truth you need to understand what your AI is actually doing in production, not what you expect it to do in tests.

Traces and spans

A trace represents the whole workflow. A span represents one individual step inside that workflow — a single LLM call, a tool invocation, a retrieval operation, or any other discrete unit of work. A simple chatbot request might produce one trace with one span. A RAG pipeline might produce one trace with four spans: embed the query, retrieve documents, re-rank results, and generate the response. The trace holds the end-to-end picture; spans let you isolate latency, cost, and correctness at each step.

What gets captured

Every trace and span records:

Field	What it contains
Input	The full prompt or query sent to the model
Output	The full response returned
Tokens	Input token count, output token count
Latency	Duration in milliseconds
Cost	Estimated cost based on model pricing
Model	Model name, version, and provider
Metadata	User ID, session ID, feature flags, custom tags
Errors	Stack traces and error messages when calls fail

Instrumenting your application

Use the SDK to create traces and attach spans wherever your application calls an LLM or performs a step you want to observe.

import { AIEvalClient } from '@evalgate/sdk';

const client = AIEvalClient.init({
  apiKey: process.env.EVALGATE_API_KEY
});

// Create a trace for the full workflow
const trace = await client.traces.create({
  name: 'Customer Support Query',
  traceId: 'trace-' + Date.now(),
  metadata: { userId: 'user_123', sessionId: 'session_456' }
});

// Add a span for the LLM call
await client.traces.createSpan(trace.id, {
  name: 'LLM Call',
  type: 'llm',
  input: userQuery,
  output: response,
  metadata: { model: 'gpt-5.5', tokens: 150 }
});

from evalgate_sdk import AIEvalClient
from evalgate_sdk.types import CreateTraceParams, CreateSpanParams
import time

client = AIEvalClient.init()  # reads EVALGATE_API_KEY env var

# Create a trace for the full workflow
trace = await client.traces.create(CreateTraceParams(
    name='Customer Support Query',
    trace_id=f'trace-{int(time.time() * 1000)}',
    metadata={'userId': 'user_123', 'sessionId': 'session_456'}
))

# Add a span for the LLM call
await client.traces.create_span(trace.id, CreateSpanParams(
    name='LLM Call',
    type='llm',
    input=user_query,
    output=response,
    metadata={'model': 'gpt-5.5', 'tokens': 150}
))

Multi-step workflows

For pipelines with multiple LLM calls or tool steps, attach one span per step to the same trace. This lets you see the full timeline and pinpoint exactly where latency or quality problems occur.

import { WorkflowTracer, traceWorkflowStep } from '@evalgate/sdk';

const tracer = new WorkflowTracer(client);

await tracer.startWorkflow('RAG Pipeline');

const embedding = await traceWorkflowStep(tracer, 'embed-query', async () => {
  return await openai.embeddings.create({...});
});

const docs = await traceWorkflowStep(tracer, 'retrieve-docs', async () => {
  return await vectorDb.search(embedding);
});

const response = await traceWorkflowStep(tracer, 'generate-response', async () => {
  return await openai.chat.completions.create({...});
});

await tracer.endWorkflow({ answer: response }, 'completed');

from evalgate_sdk import WorkflowTracer
from evalgate_sdk.workflows import trace_workflow_step

tracer = WorkflowTracer(client)

await tracer.start_workflow("RAG Pipeline")

embedding = await trace_workflow_step(
    tracer, "embed-query",
    lambda: openai.embeddings.create(...)
)

docs = await trace_workflow_step(
    tracer, "retrieve-docs",
    lambda: vector_db.search(embedding)
)

response = await trace_workflow_step(
    tracer, "generate-response",
    lambda: openai.chat.completions.create(...)
)

await tracer.end_workflow({"answer": str(response)})

Asymmetric sampling

For high-volume applications, analyzing every successful request can be expensive and unnecessary. EvalGate uses asymmetric sampling for server-side failure analysis by default:

10% of successful traces are queued for analysis
100% of error traces and thumbs-down feedback traces are queued for analysis

This controls analysis work after traces are ingested. SDK collector helpers send traces by default unless you configure client-side sampling.

Always send full traces during development and staging. Reserve client-side sampling for production once you have a baseline sense of your traffic patterns and failure rate.

Enriching traces with metadata

Traces become much more useful when they carry business context alongside the model inputs and outputs. Attach metadata at trace creation time to enable filtering, grouping, and alerting in the dashboard.

await tracer.startWorkflow('content-generation', undefined, {
  userId: user.id,
  contentType: 'blog-post',
  targetAudience: 'developers',
  keywords: ['AI', 'evaluation', 'testing']
});

await tracer.start_workflow("content-generation", metadata={
    "userId": user.id,
    "contentType": "blog-post",
    "targetAudience": "developers",
    "keywords": ["AI", "evaluation", "testing"]
})

Never log sensitive PII inside trace metadata without proper anonymization. By default, EvalGate blocks external judge calls that carry detected PII, but raw metadata is stored as-is. Apply anonymization at the application layer before calling client.traces.create or startWorkflow.

Viewing traces in the dashboard

Once your application is instrumented, open the Traces page in your EvalGate dashboard to:

Search and filter traces by metadata, tags, model, or time range
View detailed timelines showing nested spans and their durations
Analyze token usage and cost breakdowns per step
Inspect full input and output text for any span
Debug failures with complete error stack traces
Identify performance bottlenecks across the workflow

From traces to evaluations

The most important thing you can do with a traced failure is convert it into a permanent test case. EvalGate’s label command walks you through every captured trace and lets you mark each one pass or fail, assign a failure mode, and add it to your golden dataset.

# Label production traces interactively
npx @evalgate/sdk label
# Arrow-key menu, u to undo, Ctrl-C saves progress

# See failure-mode frequency across all labeled traces
npx @evalgate/sdk analyze

Once labeled and promoted into an eval suite, those traces can become evaluation cases that run on every code change. Real production failures become regression coverage after review or promotion. See Evaluations for how to build test suites from your labeled traces, and The trace → eval → gate workflow for how the full loop fits together.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Traces

Traces: visibility into LLM behavior

Traces and spans

What gets captured

Instrumenting your application

Multi-step workflows

Asymmetric sampling

Enriching traces with metadata

Viewing traces in the dashboard

From traces to evaluations

​Traces: visibility into LLM behavior

​Traces and spans

​What gets captured

​Instrumenting your application

​Multi-step workflows

​Asymmetric sampling

​Enriching traces with metadata

​Viewing traces in the dashboard

​From traces to evaluations

Traces: visibility into LLM behavior

Traces and spans

What gets captured

Instrumenting your application

Multi-step workflows

Asymmetric sampling

Enriching traces with metadata

Viewing traces in the dashboard

From traces to evaluations