Automatically capture AI interactions

Optional setup for saving complete production interactions, timing, token usage, retrieval, and tool details.

You do not need this guide to write evals or run a local gate. Start with preferred and undesired response examples when you want the shortest path. When you need automatic production capture, EvalGate saves one complete user interaction as a trace. A model call, retrieval, or tool action inside that interaction is a span. Those records become searchable and can include timing, token, and cost details. This guide adds that optional capture to an application. OpenTelemetry collectors are an interoperability option, not a prerequisite.

Install the SDK

npm install @evalgate/sdk
# or
yarn add @evalgate/sdk
# or
pnpm add @evalgate/sdk

pip install evalgate-sdk

Set environment variables

Create a .env file in your project root and add your credentials:

.env

EVALGATE_API_KEY=sk_test_your_api_key_here
EVALGATE_ORGANIZATION_ID=00000000-0000-4000-8000-000000000001

Get your API key from the Developer Dashboard.

Initialize the client and tracer

import { AIEvalClient, WorkflowTracer } from '@evalgate/sdk'

const client = new AIEvalClient({
  apiKey: process.env.EVALGATE_API_KEY
})

const tracer = new WorkflowTracer(client)

import os
from evalgate_sdk import AIEvalClient, WorkflowTracer

client = AIEvalClient(api_key=os.environ["EVALGATE_API_KEY"])
tracer = WorkflowTracer(client)

Create traces

A trace represents one logical operation — a user query, a support ticket, or a content generation request. Create a trace with a descriptive name and attach metadata that will help you filter it later:

const trace = await client.traces.create({
  name: 'Customer Support Query',
  traceId: 'trace-' + Date.now(),
  metadata: {
    userId: 'user_123',
    sessionId: 'session_456'
  }
})

import time
from evalgate_sdk.types import CreateTraceParams

trace = await client.traces.create(CreateTraceParams(
    name="Customer Support Query",
    trace_id=f"trace-{int(time.time() * 1000)}",
    metadata={"userId": "user_123", "sessionId": "session_456"}
))

Add spans

Spans represent individual steps within a trace — an LLM call, a vector search, or a function execution. Attach each span to its parent trace:

const span = await client.traces.createSpan(trace.id, {
  name: 'LLM Call',
  spanId: 'span-' + Date.now(),
  type: 'llm',
  startTime: new Date().toISOString(),
  input: userQuery,
  output: response,
  metadata: {
    model: 'gpt-5.5',
    tokens: 150
  }
})

from evalgate_sdk.types import CreateSpanParams

span = await client.traces.create_span(trace.id, CreateSpanParams(
    name="LLM Call",
    span_id=f"span-{int(time.time() * 1000)}",
    type="llm",
    input=user_query,
    output=response,
    metadata={"model": "gpt-5.5", "tokens": 150}
))

Nested spans for multi-step workflows

For pipelines with multiple sequential steps — like a RAG workflow with embedding, retrieval, and generation — use traceWorkflowStep to create properly nested spans automatically:

import { traceWorkflowStep } from '@evalgate/sdk'

await tracer.startWorkflow('RAG Pipeline')

const embedding = await traceWorkflowStep(tracer, 'embed-query', async () => {
  return await openai.embeddings.create({ /* ... */ })
})

const docs = await traceWorkflowStep(tracer, 'retrieve-docs', async () => {
  return await vectorDb.search(embedding)
})

const response = await traceWorkflowStep(tracer, 'generate-response', async () => {
  return await openai.chat.completions.create({ /* ... */ })
})

await tracer.endWorkflow({ status: 'success' })

from evalgate_sdk.workflows import trace_workflow_step

await tracer.start_workflow("RAG Pipeline")

embedding = await trace_workflow_step(
    tracer, "embed-query",
    lambda: openai.embeddings.create(...)
)

docs = await trace_workflow_step(
    tracer, "retrieve-docs",
    lambda: vector_db.search(embedding)
)

response = await trace_workflow_step(
    tracer, "generate-response",
    lambda: openai.chat.completions.create(...)
)

await tracer.end_workflow({"status": "success"})

Adding custom metadata

Attach business context to traces to make them filterable and useful for debugging:

await tracer.startWorkflow('content-generation', undefined, {
  userId: user.id,
  contentType: 'blog-post',
  targetAudience: 'developers',
  keywords: ['AI', 'evaluation', 'testing']
})

const span = await tracer.startAgentSpan('ContentAgent', { input: '...' })
// Your LLM call here
await tracer.endAgentSpan(span, { result: '...' })

await tracer.endWorkflow({ status: 'success' })

await tracer.start_workflow("content-generation", metadata={
    "userId": user.id,
    "contentType": "blog-post",
    "targetAudience": "developers",
    "keywords": ["AI", "evaluation", "testing"]
})

span = await tracer.start_agent_span("ContentAgent", {"input": "..."})
# Your LLM call here
await tracer.end_agent_span(span, {"result": "..."})

await tracer.end_workflow({"status": "success"})

What a trace can hold

Traces and spans store what you or an SDK integration (such as traceOpenAI) send:

Field	Description
Input / Output	Prompts and model responses
Timing	Start time, end time, and duration
Tokens	Provider token usage, when captured
Model	Model name and parameters
Metadata	User ID, session ID, and any custom tags you attach
Errors	Error status and messages on failure

Viewing traces

Once your application is instrumented, open the Traces page in your dashboard to:

Search and filter traces by metadata, tags, or time range
View detailed timelines showing nested spans
Analyze token usage and costs across requests
Debug failures with full stack traces
Identify latency bottlenecks across pipeline steps

Best practices

Use descriptive names

Name traces after the user action, not the implementation. customer-support-query is more useful than llm-call.

Attach relevant metadata

Include userId, sessionId, environment, and feature flags so you can slice and debug traces effectively.

Sample for high volume

For high-throughput applications, sample in your application code (for example, 10–20% of requests) rather than tracing every call.

Never log PII

Anonymize or redact sensitive user data before it appears in trace inputs, outputs, or metadata fields.

Troubleshooting

Traces not appearing in the dashboard? Verify your EVALGATE_API_KEY is correct and that AIEvalClient is initialized before any traces are created. Noticing added latency? Each trace or span write is a network request. Keep explicit client.traces.create / createSpan calls off the user-facing critical path where you can; the traceOpenAI wrapper already records its trace without blocking the model response. Spans missing data? Ensure every async function inside a traceWorkflowStep callback is properly await-ed. Unawaited promises can resolve after the span closes.

Get Started

Core Concepts

Guides

SDK Reference

Platform

Tracing setup

Automatically capture AI interactions

Install the SDK

Set environment variables

Initialize the client and tracer

Create traces

Add spans

Nested spans for multi-step workflows

Adding custom metadata

What a trace can hold

Viewing traces

Best practices

Use descriptive names

Attach relevant metadata

Sample for high volume

Never log PII

Troubleshooting

​Automatically capture AI interactions

​Install the SDK

​Set environment variables

​Initialize the client and tracer

​Create traces

​Add spans

​Nested spans for multi-step workflows

​Adding custom metadata

​What a trace can hold

​Viewing traces

​Best practices

Use descriptive names

Attach relevant metadata

Sample for high volume

Never log PII

​Troubleshooting

Automatically capture AI interactions

Install the SDK

Set environment variables

Initialize the client and tracer

Create traces

Add spans

Nested spans for multi-step workflows

Adding custom metadata

What a trace can hold

Viewing traces

Best practices

Troubleshooting