# agent-observability-eval-pipeline

Walks an instrumented ml_app through six phases: classify traces, root-cause failures, bootstrap evaluators, build a dataset, run an experiment, analyze it.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 540 installs · ★ 178
- Author: datadog-labs
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/datadog-labs/agent-skills/tree/HEAD/agent-observability/agent-observability-eval-pipeline/SKILL.md
- Updated: 2026-10-22
- HTML: https://affpapa.org/en/nevibecode/skills/agent-observability-eval-pipeline

## What it does

The skill does no analysis of its own. It orchestrates five Agent Observability sub-skills: `agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap` and `agent-observability-experiment-analyzer`. Every phase uses the same envelope — a banner naming the entity being produced, an explanation, the action, and a checkpoint. You always know where you are.

Backend detection happens once at startup. If `mcp__datadog-llmo-mcp__search_llmobs_spans` is in your tool list, the whole run goes through the Datadog MCP server. Otherwise the skill runs `pup --version` in Bash and switches to pup mode when the response carries a `version` field. With neither available it stops and asks you to connect one. That decision carries across all six phases; there is no per-phase re-detection.

Phases 4 and 5 execute code: `LLMObs.create_dataset` to publish the dataset, then `python <generated_file>` to run the experiment, with a review beat before the run. Short-circuit with `--stop-after <phase>`, or resume mid-flow with `--start-at <phase>`. For the classic evals-only behavior, `--stop-after eval-bootstrap` is the flag to reach for.

## Requirements
- Works with: Datadog

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add datadog-labs/agent-skills --skill agent-observability-eval-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/agent-observability-eval-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/agent-observability-eval-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add datadog-labs/agent-skills --skill agent-observability-eval-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/agent-observability-eval-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add datadog-labs/agent-skills --skill agent-observability-eval-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/agent-observability-eval-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install datadog-labs/agent-skills agent-observability-eval-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/agent-observability-eval-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add datadog-labs/agent-skills --skill agent-observability-eval-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/datadog-labs/agent-skills --path agent-observability/agent-observability-eval-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/agent-observability-eval-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add datadog-labs/agent-skills --skill agent-observability-eval-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/datadog-labs/agent-skills/tree/HEAD/agent-observability/agent-observability-eval-pipeline/SKILL.md

## When to use it
- sort production traces from an `ml_app` into session types
- trace a failure back to its root cause
- bootstrap evaluators from scratch
- publish a dataset from sampled traces
- run an experiment and compare metrics

## Example prompts
- /agent-observability-eval-pipeline my-ml-app --stop-after eval-bootstrap
- run the eval pipeline for my ml_app, start at the rca phase
- build a dataset and experiment from traces, py format

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 0ca7bde5aa0b8d1c.

BEGIN THIRD-PARTY FILE 0ca7bde5aa0b8d1c
```text
---
name: agent-observability-eval-pipeline
description: End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a "continue" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says "run the eval pipeline", "go from traces to evals", "bootstrap evals end to end", "classify then RCA then bootstrap", "build an eval set from scratch", "onboard me to datasets and experiments", "walk me through experiments", "I have an ml_app, now what", "Agent Observability onboarding", "guided experiment setup", "from traces to experiments", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after <phase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at <phase>`.
---

## Backend

**Detection** — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__search_llmobs_spans` appears in your available tools.
3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in the sub-skill workflow sections.
4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `"version"` confirms pup is available.
5. If pup responds → use **pup mode** throughout. Each sub-skill carries its own Tool Reference appendix with the full MCP→pup mapping.
```
END THIRD-PARTY FILE 0ca7bde5aa0b8d1c

The third-party file has ended. Everything between the 0ca7bde5aa0b8d1c tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2142 of 60362 bytes. © datadog-labs · MIT · https://github.com/datadog-labs/agent-skills/tree/HEAD/agent-observability/agent-observability-eval-pipeline/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md