agent-observability-eval-pipeline — Claude Code skill: traces to evals

---
name: agent-observability-eval-pipeline
description: End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a "continue" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says "run the eval pipeline", "go from traces to evals", "bootstrap evals end to end", "classify then RCA then bootstrap", "build an eval set from scratch", "onboard me to datasets and experiments", "walk me through experiments", "I have an ml_app, now what", "Agent Observability onboarding", "guided experiment setup", "from traces to experiments", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after <phase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at <phase>`.
---

## Backend

**Detection** — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__search_llmobs_spans` appears in your available tools.
3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in the sub-skill workflow sections.
4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `"version"` confirms pup is available.
5. If pup responds → use **pup mode** throughout. Each sub-skill carries its own Tool Reference appendix with the full MCP→pup mapping.
6. If neither is available → stop and tell the user:
   > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (`claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/v1/mcp?toolsets=llmobs'`) or install pup."

`--backend pup` is accepted anywhere in the invocation arguments. Strip it from args before passing to sub-skills, but carry the pup-mode decision forward — every sub-skill must also operate in pup mode for the entire pipeline run.

**Sub-skill backend propagation**: The backend detected at startup applies to all sub-skills invoked across the six phases. Do not re-detect per phase. Announce once at startup:
- MCP mode: "(Running in MCP mode — all features available.)"
- pup mode: "(Running in pup mode — pup commands used throughout. All features available.)"

**Invocation ID:** At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation.

**Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:agent-observability-eval-pipeline[<inv_id>] — ` followed by a description of why the tool is being called. On the **first MCP tool call only**, use `skill:agent-observability-eval-pipeline:start[<inv_id>] — ` instead (note the `:start` suffix). Example first call: `skill:agent-observability-eval-pipeline:start[3a9f1c2b] — Precheck: verify ml_app has traces in the last 7 days`

---

# Agent Observability Eval Pipeline — Classify → RCA → Eval Bootstrap → Dataset → Experiment → Analyze

A deterministic, six-phase guided pipeline for an already-instrumented `ml_app` owner. Each phase has the same envelope — a banner that names the entity being produced, an explanation of its purpose, the action (a sub-skill call or a small executable step), and a checkpoint. **You always know where you are.**

```
[Precheck] verify ml_app, project, backend, credentials, output dir
   ↓
[Phase 1: Classify ml_app traces]        entity: ml_app, trace, span
   ↓
[Phase 2: Root cause analysis]           entity: failure mode, root cause
   ↓
[Phase 3: Bootstrap evaluators]          entity: evaluator, LLM judge
   ↓                                     (stop here with --stop-after eval-bootstrap
   ↓                                      for the classic eval-pipeline behavior)
[Phase 4: Create + publish dataset]      entity: dataset record, published dataset
   ↓                                     (executes: LLMObs.create_dataset)
[Phase 5: Generate + run experiment]     entity: experiment, task, evaluator, run
   ↓                                     (executes: python <generated_file>)
   ↓                                     (in-phase review beat before run)
[Phase 6: Analyze experiment]            entity: metric, comparison, recommendation
```

This skill is **pure orchestration plus pedagogy** — no new analytical logic. The work happens inside the sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-bootstrap`, `agent-observability-experiment-analyzer`). What this skill adds is the deterministic envelope: every phase has the same shape, the same checkpoint contract, and the same entity-explanation banner — so the user gets a consistent, narrated experience regardless of how they phrased the original request.

## Usage

```
/agent-observability-eval-pipeline <ml_app> [--project-name <name>] [--timeframe <window>] [--trace-limit <N>]
                                [--format py|ipynb] [--evaluator-style function|class|remote]
                                [--offline-evaluators | --online-evaluators | --data-only]
                                [--start-at classify|rca|eval-bootstrap|dataset|experiment|analyze]
                                [--stop-after classify|rca|eval-bootstrap|dataset|experiment|analyze]
                                [--classification-summary <path>] [--rca-report <path>]
                                [--dataset-file <path>] [--dataset-name <name>]
                                [--experiment-file <path>] [--experiment-id <uuid> | --experiment-url <url>]
                                [--app-root <path>] [--env-file <path>] [--output-dir <dir>]
                                [--backend pup]
```

Arguments: $ARGUMENTS

### Inputs

| Input | Required | Default | Description |
|-------|----------|---------|-------------|
| `ml_app` | Yes | — | The instrumented LLM app to onboard / evaluate against. The precheck verifies it has recent traces. |
| `--project-name` | No | derived from `pyproject.toml` / `setup.cfg` / `setup.py` / `package.json` / cwd (same order as `agent-observability-experiment-bootstrap`); falls back to `experiment-sdk-default` | The Datadog **project** the pipeline writes datasets and experiments into. The SDK lazily creates the project on first use via `LLMObs.enable(project_name=...)`. Surface this in the Precheck so the user can confirm before anything is created. |
| `--timeframe` | No | `now-7d` | Lookback window for Phase 1 classification and Phase 4 dataset sampling. |
| `--trace-limit` | No | `20` | Sampling cap for Phase 4. Phase 1 internally uses `min(20, --trace-limit)` for the classification sample. |
| `--format` | No | `py` | Passed to `agent-observability-experiment-bootstrap` in Phase 5: `py` (script) or `ipynb` (Jupyter notebook). |
| `--evaluator-style` | No | `function` | Passed to `agent-observability-eval-bootstrap` (Phase 3) and `agent-observability-experiment-bootstrap` (Phase 5): `function`, `class`, or `remote`. |
| `--offline-evaluators` | No | on (default) | Phase 3: emit a Python SDK evaluator suite (BaseEvaluator / LLMJudge classes) that runs inside an experiment against a dataset. Maps internally to `agent-observability-eval-bootstrap` `sdk_code` mode. |
| `--online-evaluators` | No | off | Phase 3: publish online LLM-judge evaluators directly to Datadog (created as disabled drafts; enable in the UI). Online evaluators run on production spans as they're emitted. Maps internally to `agent-observability-eval-bootstrap` `publish` mode (was `--publish`). |
| `--data-only` | No | off | Phase 3: emit a local data blob only — no executable evaluator code or online publish. At Phase 3 entry the skill **prompts** the user to pick one of: (a) a `DatasetRecordRaw[]` JSON suitable for experiment use (maps internally to `agent-observability-eval-bootstrap --emit-dataset`), or (b) a framework-agnostic JSON evaluator spec for local analysis (maps internally to `agent-observability-eval-bootstrap` `data_only` mode). |
| `--stop-after <phase>` | No | `analyze` (run everything) | Stop after the named phase completes. `classify` = Phase 1 only. `rca` = through Phase 2. `eval-bootstrap` = through Phase 3 (matches the classic eval-pipeline). `dataset` = through Phase 4 (dataset created + published). `experiment` = through Phase 5 (experiment generated + run). `analyze` = all six phases (default). |
| `--start-at <phase>` | No | `classify` (start at the top) | Skip earlier phases and start at the named phase. Same vocabulary as `--stop-after`. The skill auto-loads any required prior-phase artifacts from `<output-dir>/state/` (see "State persistence and entry/exit" section). For phases that need an artifact the auto-load can't find, supply it via one of the override flags below. Combinable with `--stop-after` to run a contiguous slice of the pipeline. |
| `--classification-summary <path>` | No | auto-loaded from `<output-dir>/state/01-classification.md` if `--start-at rca` or later | Override the Phase 1 output that Phase 2 consumes. Useful when the prior state file is missing or you want to point at a hand-edited version. |
| `--rca-report <path>` | No | auto-loaded from `<output-dir>/state/02-rca-report.md` if `--start-at eval-bootstrap` or later | Override the Phase 2 output that Phase 3 consumes. |
| `--dataset-file <path>` | No | auto-loaded from `<output-dir>/state/04-published-dataset.json`'s `dataset_file` field (or the most recent `<output-dir>/dataset_<ml_app>_*.json`) | The local `DatasetRecordRaw[]` JSON. Used by Phase 4's publish sub-step when re-publishing without re-sampling. |
| `--dataset-name <name>` | No | auto-loaded from `<output-dir>/state/04-published-dataset.json` if `--start-at experiment` | The name of a published Datadog dataset that Phase 5 wires the experiment to. |
| `--experiment-file <path>` | No | auto-loaded from `<output-dir>/state/05-experiment-run.json`'s `experiment_file` field if `--start-at experiment` and the file already exists | The generated experiment file. When present, Phase 5 skips the codegen sub-step (5a) and goes straight to the review beat (5b) → run (5c). |
| `--experiment-id <uuid>` | No | auto-loaded from `<output-dir>/state/05-experiment-run.json` if `--start-at analyze` | The Datadog experiment ID Phase 6 analyzes. Mutually exclusive with `--experiment-url`. |
| `--experiment-url <url>` | No | auto-loaded as above | Alternative to `--experiment-id`. The skill parses the trailing UUID out of the URL. |
| `--app-root` | No | resolved from cwd / `pyproject.toml` etc. | Restricts `agent-observability-experiment-bootstrap`'s task-function introspection to this directory tree. |
| `--env-file` | No | none (auto-discovery walks standard locations) | Explicit `.env` path for credential loading. Surfaced in the Precheck and baked into the generated experiment as `ENV_FILE_OVERRIDE`. |
| `--output-dir` | No | `./experiments` | Where the dataset JSON, publish script, and generated experiment file are written. |
| `--backend` | No | auto-detect | `pup` forces pup mode regardless of MCP availability. |

If `ml_app` is not provided, ask the user before proceeding. The three evaluator-output flags (`--offline-evaluators`, `--online-evaluators`, `--data-only`) are mutually exclusive — error out if more than one is set. If none are set, `--offline-evaluators` is the default. The legacy flag names `--publish` and `--sdk-code` are still accepted as aliases for backward compatibility but map to the new names in all output.

---

## Precheck

Before Phase 1, run a single short verification pass — do **not** announce a "Phase" banner for this; it's plumbing. Output a one-block precheck summary, then move directly to Phase 1.

1. **Backend** — already detected at the top of this skill. Note the chosen backend in the precheck output so the user can confirm.

2. **ml_app has recent traces** — call `search_llmobs_spans(query="@ml_app:\"<ml_app>\"", root_spans_only=true, limit=1, from="<timeframe>")` (MCP) or the pup equivalent. If the result is empty, stop and tell the user the precheck failed — there is nothing to evaluate against — and suggest widening `--timeframe` or confirming the ml_app name.

3. **Resolve `project_name`** — if `--project-name <name>` was supplied, use it verbatim. Otherwise derive using the same resolution order as `agent-observability-experiment-bootstrap` (Workflow step 1): `pyproject.toml` → `setup.cfg` → `setup.py` → `package.json` → cwd basename (slugified). Final value is `experiment-<service-name>`; fall back to `experiment-sdk-default` if nothing resolves and emit a warning telling the user to set `--project-name` explicitly.

   **Project creation semantics**: the project is created lazily by the Datadog SDK the first time `LLMObs.enable(project_name=...)` is called against the org (in Phase 4's publish script, and again in Phase 5's generated experiment). The user does not need to pre-create anything in the UI. Surface the chosen project name in the Precheck output so the user can override before Phase 4 if it isn't what they wanted.

4. **Ensure `--output-dir` and `<output-dir>/state/` exist** — `mkdir -p <output-dir>/state` via Bash. Cheap. The `state/` subdirectory is where phase outputs get persisted (see "State persistence and entry/exit" below).

5. **Resolve credentials.** Walk the discovery order below to find Datadog credentials before Phase 4 needs them — failing late at the publish step is bad UX. Read-only at this stage: do NOT write any new files. Do NOT print secret values to the user; only report which file was loaded and which keys were resolved.

   **Discovery order** (first hit per variable wins; shell env vars always override files):
   1. **`--env-file <path>`** override if supplied — always tried first.
   2. **Current shell environment** (`os.environ`) — already-exported `DD_API_KEY` / `DD_APPLICATION_KEY` / `DD_APP_KEY` / `DD_SITE` take precedence over file values. If all required keys are already present, skip file loading entirely.
   3. **`<output-dir>/.env`** — if the user previously ran the skill and dropped a `.env` next to past artifacts, prefer that.
   4. **`<app-root>/.env`** — the resolved `pyproject.toml` / `setup.cfg` / `setup.py` / `package.json` directory (or cwd if none).
   5. **`<app-root>/.env.local`** — git-ignored local override convention.
   6. **`<cwd>/.env`** — fallback if cwd differs from app-root.
   7. **Parent walk**: from cwd, walk up directory by directory looking for `.env` until reaching `/` or the user's home directory. Stop at the first hit.
   8. **`~/.datadog/credentials`** — Datadog's well-known per-user credentials file, if present.

   For each file checked, parse line-by-line: skip blanks / comment lines (`#`) / malformed lines (no `=`). Strip a leading `export ` if present (so `.envrc`-style files work). Split on the first `=`. Strip surrounding quotes on the value. Only set a variable that is not already in `os.environ` — never overwrite the shell.

   **Required keys**: `DD_API_KEY` AND (`DD_APPLICATION_KEY` OR `DD_APP_KEY`). `DD_SITE` is optional (defaults to `datadoghq.com`). Provider keys (`OPENAI_API_KEY`, `ANTHROPIC_API_KEY`, etc.) are validated later in Phases 6/7 against the introspected task — not here.

   **If all required keys resolved** — record which file(s) were loaded and emit a single-line summary in the Precheck block. Continue.

   **If required keys NOT found after walking every location** — stop and prompt the user with a clear, actionable message:

   > "Datadog credentials were not found in your shell env or any discovered `.env` file. Two options before continuing:
   > - **Export in your shell**: `export DD_API_KEY=…` and `export DD_APPLICATION_KEY=…` (also `DD_SITE=…` if non-default), then re-invoke this skill.
   > - **Drop a `.env`** at `<app-root>/.env` with `DD_API_KEY=…` and `DD_APPLICATION_KEY=…` on separate lines, then re-invoke.
   >
   > Make sure `.env` is in your `.gitignore` before committing."

   Do **not** offer to create the `.env` file for the user — secrets-on-disk decisions belong to the user, not the skill.

Output the precheck summary, then start Phase 1:

```
## Precheck

- Backend: <MCP | pup>
- ml_app `<ml_app>` has traces in <timeframe>: yes (<sample_count> root spans found)
- Project name: `<project_name>` (created lazily on first LLMObs.enable() call in Phase 4)
- Output dir: `<output-dir>` (created)
- Credentials: <one of:
    "loaded from shell env (DD_API_KEY, DD_APPLICATION_KEY, DD_SITE)"
  | "loaded from <relative path to .env file> (DD_API_KEY, DD_APPLICATION_KEY[, DD_SITE])"
  | "shell env + <relative path>: keys resolved from both (shell overrode file for <list>)"
  >
- Stop-after: <phase from --stop-after, default `analyze`>

Starting Phase 1 of 6.
```

The exact list of keys in the credentials parenthetical reflects what was actually discovered (so the user can verify nothing surprising was loaded). Never print the values.

---

## Phase Template

Every phase below uses this exact template. Do not deviate — the deterministic envelope is what makes the pipeline experience consistent across invocations.

```
## Phase N of 6: <Title>

**You are here.** Phase N of 6 — <one-line position summary>.

**What this phase produces**: <Entity name>
**What a <entity> is**: <2-3 sentence definition tailored to this phase>
**Why it matters**: <1 sentence on why the user needs this>

→ Action: <invoke <sub-skill> | execute <script>>

<full sub-skill output OR execution log reproduced here — do NOT summarize or truncate>

<after the action completes successfully, Write the phase output to
`<output-dir>/state/0N-<name>.{md,json}` per the State persistence contract — see
"State persistence and entry/exit" section. This happens BEFORE the checkpoint
so the file is on disk even if the user types `stop` next.>

---

### Checkpoint <N>

<concise summary of what was produced, where it lives (path / Datadog URL), and any caveats>

Before I continue to Phase N+1 (<next title>):
- <2–3 phase-specific review prompts the user can answer>

Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments.
```

**Never auto-advance.** Always pause at the checkpoint and wait for explicit user input. The whole point of this skill is determinism — that includes determinism over *when* the user moves on.

If the current phase matches the value of `--stop-after`, replace the checkpoint prompt with a **Stop summary** (see "Stop-after handling" at the bottom of this file).

---

## Phase 1: Classify ml_app traces

**Entity**: `ml_app`, `trace`, `span`.

**Pedagogy banner** (use verbatim, adapted only to the actual ml_app name):

> **What an ml_app is**: a logical LLM application — a name you tag spans with when instrumenting (`ml_app=<name>`). It groups all production traces and evaluator runs that belong to the same product surface. Every dataset, experiment, and evaluator you create later targets this scope.
>
> **What a trace is**: one end-to-end execution of your ml_app — typically the agent loop for a single user request, made up of one or more spans (LLM calls, tool calls, retrievals).
>
> **Why this phase matters**: before you root-cause failures (Phase 2), bootstrap evaluators (Phase 3), or curate a dataset (Phase 4), you want a quick read on what your app actually does in production and where its current failure modes are. Classification gives you that signal in one pass.

**Trace pool preview (MANDATORY).** Before invoking the sub-skill, build and surface a Datadog Traces UI link that opens the **exact set of root spans Phase 4 will sample from**. This lets the user eyeball the pool, spot outliers, or adjust `--timeframe` before any classification work runs. The link must match the filter that `agent-observability-eval-bootstrap --emit-dataset` uses in Phase 4 (see `agent-observability/agent-observability-eval-bootstrap/SKILL.md` → Phase 3D → Sampling): `@ml_app:"<ml_app>" @status:ok`, root spans only.

URL construction rules:

1. **Host**: prepend `app.` to `DD_SITE`. If `DD_SITE` is unset, default to `app.datadoghq.com`.
   - `datadoghq.com` → `app.datadoghq.com`
   - `datadoghq.eu` → `app.datadoghq.eu`
   - `us3.datadoghq.com` → `app.us3.datadoghq.com`
   - `us5.datadoghq.com` → `app.us5.datadoghq.com`
   - `ap1.datadoghq.com` → `app.ap1.datadoghq.com`
   - `ap2.datadoghq.com` → `app.ap2.datadoghq.com`
   - `datad0g.com` → `app.datad0g.com` (staging)
2. **Path**: `/llm/traces` (the Agent Observability Traces explorer).
3. **Query string**: URL-encode `@ml_app:"<ml_app>" @status:ok @parent_id:undefined` and bind to `query=`.
4. **Time window**: convert `--timeframe` to absolute epoch milliseconds (`now - <duration in ms>` for the start, `now` for the end) and bind to `start=` / `end=`.
5. **Optional**: append `&paused=true` so the UI does not live-stream the result on open.

Surface the link immediately before the Action block:

```
**Trace pool being analyzed** ({timeframe}, filter `@ml_app:"<ml_app>" @status:ok`, root spans only):

  <full URL>

Phase 1 will classify the first {min(20, trace-limit)} of these for orientation. Phase 4 will sample up to {trace-limit} from the same pool for the dataset. Click through if you want to see the pool before either runs, or adjust `--timeframe` and re-invoke if the window looks off.
```

**Action**: Follow the **`agent-observability-session-classify`** skill in **ml_app mode**, using:
- `ml_app` = the provided ml_app
- `timeframe` = the provided timeframe
- `sample_limit` = `min(20, trace-limit)` — keep this fast; Phase 4 will do the bigger sample

Run the complete ml_app mode workflow as defined in that skill (Steps M1 through M3). **Output the full classification output** (all compact per-unit blocks plus the final `# Session Classification Summary`) — do not summarize or truncate. Downstream Phase 2's RCA depends on the full text being in context.

### Checkpoint 1

After the `# Session Classification Summary` is output, present:

```
## Phase 1 complete — you've seen what `<ml_app>` does in production

[verdict distribution table from session-classify]
[failure mode frequency table from session-classify]

**Trace pool for Phase 4's dataset sample**: <full URL from the Trace pool preview above>
(same filter / same timeframe — open it now if you want to scan for traces you'd rather exclude)

Next up — Phase 2 will diagnose *why* the failing traces are failing.

Before I continue:
- Do these failure patterns look right?
- Any traces you'd like to exclude from the dataset sample in Phase 4? (Paste trace IDs from the link above and I'll drop them.)
- Any quality dimension you already know you want to measure later?

Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments.
```

Wait for explicit user confirmation. If the user excludes specific traces, mark them as "excluded by user" — drop them from Phase 2's failure bucket and from Phase 4's sampling. Do NOT re-classify.

---

## Phase 2: Root cause analysis

**Entity**: `failure mode`, `root cause`.

**Pedagogy banner**:

> **What a failure mode is**: a recurring pattern in *how* your app fails — e.g. "the model hallucinates citation URLs", "the agent forgets state across tool calls", "the retrieval returns irrelevant chunks". Each failure mode has one or more *root causes* (system prompt deficiency, tool gap, retrieval miss, etc.).
>
> **Why this phase matters**: an evaluator that scores generic "is this response good?" misses the specific things going wrong in your app. RCA lets Phase 3 propose evaluators that target your *actual* failure modes — sharper signal, fewer false alarms.

**Action**: Follow the **`agent-observability-trace-rca`** skill.

The `# Session Classification Summary` from Phase 1 is in context. The skill detects it automatically via its Phase 0 Step 0S check and enters the "from classifications" path — it extracts the failure bucket, presents the Classification Overview, and proceeds directly to Phase 2 (open coding) without running its own Phase 1 span search.

Run the full workflow through Phase 6 (the compiled RCA report). **Output the full RCA report** — do not summarize. The full report must be in context for Phase 3's detection to work.

### Checkpoint 2

```
## Phase 2 complete — root causes identified

[the Phase 6 RCA report is above]

Next up — Phase 3 will bootstrap evaluators that target these failure modes.

Before I continue:
- Do these root causes look accurate?
- Any failure modes to add, remove, or reframe?
- Which root causes should the evaluators target? (Default: all of them.)

Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments.
```

Wait for explicit user confirmation. If the user adjusts the taxonomy, incorporate the changes before continuing.

---

## Phase 3: Bootstrap evaluators

**Entity**: `evaluator`, `LLM judge`.

**Pedagogy banner**:

> **What an evaluator is**: a function that grades one record's output. Returns `bool` / `float` / a structured `EvaluatorResult`. Two flavors: **code evaluators** (deterministic checks — JSON validity, regex match, length, custom Python) and **LLM-as-judge evaluators** (a model graded against a rubric — e.g. "is this response grounded in the retrieved documents?").
>
> **Operating modes** (mutually exclusive; default is `--offline-evaluators`):
>
> - **`--offline-evaluators`** *(default)*: emit a Python SDK evaluator suite (BaseEvaluator / LLMJudge classes) that runs **inside an experiment against a dataset**. Use this when you'll run experiments locally.
> - **`--online-evaluators`**: publish LLM-judge evaluators to Datadog as disabled drafts. They run **on production spans as they're emitted** once enabled. Use this when you want continuous evaluation in prod.
> - **`--data-only`**: emit a local data blob only — no executable evaluator code or online publish. Prompts you to pick between a dataset for experiment use or a local analysis blob (see "Operating mode resolution" below).
>
> **Why this phase matters**: evaluators are the contract between "this output looks fine" and "this output meets our quality bar." The bootstrapped suite is grounded in the failure taxonomy from Phase 2 — sharper than generic evaluators you'd otherwise hand-write.

**Action**: Follow the **`agent-observability-eval-bootstrap`** skill.

The RCA report from Phase 2 is in context. The skill detects the `## Failure Taxonomy` heading automatically and enters its "from RCA" path in Phase 0.

### Operating mode resolution

Before invoking the sub-skill, resolve the operating mode:

1. **`--offline-evaluators`** (default) → call `agent-observability-eval-bootstrap` with no mode flag (its `sdk_code` default).
2. **`--online-evaluators`** → call `agent-observability-eval-bootstrap --publish`.
3. **`--data-only`** → prompt the user via `AskUserQuestion`:

   > "You picked `--data-only` for Phase 3. What kind of local data blob do you want?
   >
   > - **Dataset for experiment use** — a `DatasetRecordRaw[]` JSON suitable for `LLMObs.create_dataset(records=...)`. Useful when you want to seed an experiment with the records this skill samples. (Internally calls `agent-observability-eval-bootstrap --emit-dataset <path>`.)
   > - **Local blob for analysis** — a framework-agnostic JSON evaluator spec describing what evaluators *would* be generated, without emitting Python code. Useful for inspecting evaluator coverage without running anything. (Internally calls `agent-observability-eval-bootstrap --data-only`.)"

   If the user picks "Dataset for experiment use" and the pipeline is running through Phase 4 (i.e. `--stop-after` is not `eval-bootstrap` or earlier), tell the user that Phase 4's sample step will be **skipped** in favor of this Phase 3 output — the dataset they produce here will be the one Phase 4 publishes.

Pass `--evaluator-style` through unchanged.

**The agent-observability-eval-bootstrap skill has its own mandatory proposal checkpoint** (the evaluator suite proposal before code generation). Honor it — do not skip or auto-confirm it.

### Checkpoint 3

```
## Phase 3 complete — evaluator suite ready

- Mode: `<offline-evaluators | online-evaluators | data-only (dataset-for-experiment) | data-only (analysis-blob)>`
- Output: `<path to .py / .json / dataset-record JSON / "published as drafts to Datadog">`
- Evaluators emitted: <list of names>
- Coverage: <one-liner: which failure-mode categories are now covered>

Next up — Phase 4 will sample production traces into a dataset you can run experiments against (using these evaluators or the placeholders the experiment template ships with).

If you only wanted evaluators (the classic eval-pipeline flow), this is the natural stopping point: re-invoke with `--stop-after eval-bootstrap` to formalize that as the exit.

Before I continue:
- Do the generated evaluators look right?
- Any to drop or rename before they're referenced in the experiment?

Type `continue` to proceed, `stop` to exit cleanly (state is saved), `redo` to re-run this phase, or give me adjustments....