LLM Evaluations

OpenObserve offers two evaluation modes: Online Evaluations, which score live traces and spans as they happen, and Experiments, which run scorers over a static dataset for offline testing and comparison. Both use LLM-as-a-judge scorers (your own AI provider) or external remote scoring endpoints.

Overview

Activating an eval job runs your scorers against incoming data. Scores land in the _llm_scores stream; evaluator telemetry lands in the _evaluator stream.

Enable Online Evaluations

Online Evaluations is available in the Enterprise edition and OpenObserve Cloud, and is enabled by default:

O2_EVAL_ENABLED=true

When disabled, the AI menu, Settings > LLM Providers, and Settings > GenAI Agent Mapping are hidden, and the evaluation API endpoints are not registered.

Other server settings:

Providers

A Provider stores the connection details for an LLM API. Providers live under Settings > Data & AI > LLM Providers. Click New Provider to add one.

Use OpenAI-compatible for any service that implements the OpenAI chat completions API, and vLLM for a self-hosted vLLM server (defaults to http://localhost:8000/v1/chat/completions). System One (TypeSafe Jev) is available to scorers only; it does not appear in the Playground or Experiments.

Click Test Connection to check the endpoint and key before you save. To edit or delete a provider, use the actions on its row. Scorers that use a deleted provider fail until you reassign them.

Score Configs

A Score Config defines what a score looks like and when it's healthy. Navigate to AI > Evaluate > Score Configs and click New score config.

Updating a score config creates a new version; scorers can pin to a specific version or always use the latest.

Scorers

A Scorer is the evaluation unit: a prompt template with {{variable}} placeholders, execution parameters, and an optional linked score config. Navigate to AI > Evaluate > Scorers and click New scorer.

• LLM Judge: sends the rendered template to an LLM via a provider. Configure provider, model, temperature, max tokens, timeout, and optional reasoning.
• Remote: sends the rendered template as an HTTP request to an external service. Configure endpoint, method, auth (none/bearer/basic/api_key), headers, timeout, and retries.

Use the Test button on the scorer detail page to try it with sample variable values; you'll see the score, reasoning, model, latency, and token usage.

Like score configs, scorers are versioned. Eval jobs can reference a scorer by entity ID (always latest) or pin a specific version. The Versions tab in the scorer drawer lists every version, newest first, and tags the active one. Select two versions to compare them: you see a split or unified diff of the template and a table of the fields that changed.

Eval Jobs

An Eval Job runs scorers against incoming traces at a chosen target scope: span, trace, or session. Navigate to AI > Evaluate > Eval Jobs and click New job. Saving a job checks that you can read the selected stream.

Target scope

Trace and session completion

For trace- and session-scope jobs, configure how OpenObserve decides a target is "done":

Example end signal (evaluate as soon as a span with status = "complete" arrives):

{
  "version": 2,
  "conditions": {
    "filterType": "group",
    "logicalOperator": "AND",
    "conditions": [{
      "filterType": "condition",
      "column": "status",
      "operator": "=",
      "value": "complete",
      "logicalOperator": "AND"
    }]
  }
}

Span selectors (trace scope)

For trace-scope jobs, a scorer may only need a subset of a trace's spans. A Span Selector filters, picks, and limits which spans (and fields) get sent to a given scorer; bind one to any scorer whose template uses {{ spans }}.

Input mapping and template variables

Each scorer's {{variables}} are mapped to a source in the job form's Prompt variables section, either a system-provided value or a span attribute.

For trace/session scope, OpenObserve auto-supplies these as ready-to-use template variables:

For span-scope jobs, every variable maps to a span attribute instead (e.g. input → {{gen_ai_input_messages}}).

Manual evaluation

Trigger an evaluation on demand for a specific trace, span, or session. This is useful for re-testing after changing scorers. From the trace, span, or session detail page, click Evaluate trace / Evaluate span / Evaluate session, then pick which Eval Job to run.

Job lifecycle

draft → active ⇄ paused
          ↓
       degraded → active
          ↓
       archived

• Activate: starts scoring.
• Pause: stops it temporarily.
• Resume: restarts it.
• Archive: stops it permanently.

Editing an active job's config (filters, sampling, scorers, scope) takes effect immediately. No need to pause first.

Where scores go

Scores land in the _llm_scores stream; evaluator telemetry (latency, tokens, status) lands in the _evaluator traces stream. Query both directly for debugging or dashboards.

Quality Dashboard

AI > Evaluate > Quality gives a real-time overview of evaluation health across your score configs, agents, and streams. Drill into a score config to see KPI cards, trend charts, and a runs table, filterable by scope (All/Span/Trace/Session).

When a scorer fails, click the Scorer Failures KPI card to jump to the failing evaluator traces.

Evaluation runs

The Evaluation Runs table lists individual score records: value, health, target identity, agent, and reasoning. Click a row to open its evaluator trace. Filter to all runs or unhealthy only.

Gen-AI Agents

OpenObserve auto-discovers agents from your trace telemetry (via gen_ai.agent.name/id, or your own configured attribute fields), so you can scope the Quality, Insights, and Sessions views to a single agent or All Agents.

If your telemetry uses non-standard attribute names, map them under Settings > GenAI Agent Mapping. Scorer templates can also reference {{agent_name}}, {{agent_id}}, and {{pipeline_source_stream_type}}. See LLM Monitoring for the attributes OpenObserve checks.

Experiments

For offline, batch evaluation against a versioned dataset (pinned scorers, trial counts, cost estimates, baseline comparisons), see LLM Experiments.

RBAC

Providers, Score Configs, Scorers, and Eval Jobs each have their own permissions (GET, LIST, POST, PUT, DELETE). Assign roles in IAM > Roles to control access.

API Reference

All endpoints are prefixed with /api/{org_id}.