agent-observability-eval-bootstrap — скилл для Claude Code

---
name: agent-observability-eval-bootstrap
description: Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says "bootstrap evaluators", "generate evaluators", "create evals from traces", "eval bootstrap", "write evaluators", "build eval suite", "publish evaluators", or wants to generate BaseEvaluator/LLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.
---

## Backend

**Detection** — At the start of every invocation, before taking any action, determine which backend to use:

1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4.
2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__list_llmobs_evals` appears in your available tools.
3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in this skill's workflow sections.
4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `"version"` confirms pup is available.
5. If pup responds → use **pup mode** throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.
6. If neither is available → stop and tell the user:
   > "Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (`claude mcp add --scope user --transport http datadog-llmo-mcp 'https://mcp.datadoghq.com/v1/mcp?toolsets=llmobs'`) or install pup."

`--backend pup` is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.

**pup invocation rules:**
- Invoke via Bash: `pup llm-obs <subcommand> [flags]`
- pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in `[{"type": "text", "text": "<json>"}]`).
- If pup returns an auth error, tell the user to run `pup auth login` and stop.
- Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).
- Time flags: pup accepts bare duration strings (`1h`, `7d`, `30m`) and RFC3339 timestamps. Do **not** use `now-`-prefixed strings — strip the prefix when converting from a skill `--timeframe` argument: `now-7d` → `7d`, `now-24h` → `24h`, `now-30d` → `30d`.
- `--summary` on `pup llm-obs spans search` strips payload fields to essential metadata only. Use it in bulk/search phases where content is not needed.

**Invocation ID:** At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation.

**Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:agent-observability-eval-bootstrap[<inv_id>] — ` followed by a description of why the tool is being called. On the **first MCP tool call only**, use `skill:agent-observability-eval-bootstrap:start[<inv_id>] — ` instead (note the `:start` suffix). Example first call: `skill:agent-observability-eval-bootstrap:start[3a9f1c2b] — Phase 0: map existing eval coverage for task-cruncher`

# Eval Bootstrap — Generate Evaluators from Production Traces

Given a sample of production LLM traces, analyze input/output patterns and quality dimensions, then propose a ready-to-use evaluator suite. Four output modes — **online evaluators are the default**; SDK code, the JSON spec, and the dataset-emit mode are produced on request:

- **`publish`** *(default)* — propose **online** LLM-judge evaluators, then — **only after you confirm the suite** — write them to Datadog via `create_or_update_llmobs_evaluator` as **disabled drafts** (`enabled: false`). Nothing is created until you confirm at the Phase 2 checkpoint, and nothing scores any spans until **you** enable it in the UI — the skill never auto-publishes a live evaluator. Once you enable a draft, it runs automatically on matching production spans, traces, or sessions (no dataset, no task function). The skill **auto-classifies** each proposed evaluator as **span-scoped**, **trace-scoped**, or **session-scoped** based on what the judgment requires (a per-LLM-call tone check vs. an agent goal completion that needs the whole trace vs. user satisfaction across a whole multi-trace conversation) — you accept or override the classification at that checkpoint. Session-scoped evaluators are only proposed when the app's spans carry a `session_id` (verified by a probe in Phase 1).
- **`sdk_code`** *(on request — `--sdk-code`, or ask after a publish run)* — Python `.py` file using the Datadog Evals SDK (`BaseEvaluator` / `LLMJudge`) for **offline** experiments.
- **`data_only`** *(on request — `--data-only`)* — self-contained JSON spec, framework-agnostic.
- **`emit_dataset`** *(on request — `--emit-dataset <path>`)* — sample production traces and write a `DatasetRecordRaw[]` JSON file shaped for `LLMObs.create_dataset(records=...)`. **Skips evaluator proposal and generation entirely** — this mode produces a dataset, not evaluators. Used by `agent-observability-eval-pipeline` (Phase 4) to seed an experiment dataset from production behavior.

After a publish run, if the user wants the same suite as offline code or a portable spec, they just ask — the skill regenerates the **already-confirmed** suite in `sdk_code` / `data_only` mode without re-exploring (see "On-request code generation" in Phase 3). The `emit_dataset` mode is independent of the evaluator workflow and never re-uses a prior proposal — it always re-samples traces.

## Usage

```
/eval-bootstrap <ml_app> [--timeframe <window>] [--sdk-code | --data-only | --emit-dataset <path>] [--trace-limit <N>]
```

Arguments: $ARGUMENTS

### Inputs

| Input | Required | Default | Description |
|-------|----------|---------|-------------|
| `ml_app` | Yes | — | ML application to scope traces |
| `timeframe` | No | `now-7d` | How far back to look |
| `rca_report` | No | — | Failure taxonomy from `eval-trace-rca` skill, or a free-text failure hypothesis |
| `--sdk-code` | No | off | Emit a Python SDK `.py` file for offline experiments instead of publishing online. Mutually exclusive with `--data-only` and `--emit-dataset`. |
| `--data-only` | No | off | Emit a self-contained JSON spec file instead of publishing online. Mutually exclusive with `--sdk-code` and `--emit-dataset`. |
| `--emit-dataset <path>` | No | off | **Dataset-only mode.** Sample production traces and write a `DatasetRecordRaw[]` JSON to `<path>`. Skips the evaluator workflow entirely. Mutually exclusive with `--sdk-code` and `--data-only`. |
| `--trace-limit` | No | `20` (cap `50`) | Max traces to sample in `emit_dataset` mode |

If `ml_app` is missing, ask the user before proceeding. With no mode flag, the skill defaults to **`publish`** — it proposes online evaluators and, only after you confirm, creates them as disabled drafts (it never auto-enables them). If more than one of `--sdk-code`, `--data-only`, `--emit-dataset` is supplied, error out and ask which mode the user wants.

## Available Tools

| Tool | Purpose |
|------|---------|
| `search_llmobs_spans` | Find spans by eval presence, tags, span kind, query syntax. Paginate with cursor. |
| `get_llmobs_span_details` | Metadata, evaluations (scores, labels, reasoning), and `content_info` map showing available fields + sizes. |
| `get_llmobs_span_content` | Actual content for a span field. Supports JSONPath via `path` param for targeted extraction. |
| `get_llmobs_trace` | Full trace hierarchy as span tree with span counts by kind. |
| `get_llmobs_agent_loop` | Chronological agent execution timeline (LLM calls, tool invocations, decisions). |
| `list_llmobs_evals` | List every evaluator configured for the caller's org across all ml_apps, with `enabled` status and `ml_app` per result. Call once in Phase 0 to map existing coverage before proposing new evaluators — filter the result by `ml_app` client-side. |
| `get_llmobs_evaluator` | Fetch the **full** persisted evaluator config by name (target ml_app + sampling + filter, provider, prompt template, parsing type, output schema, assessment criteria). Use in Phase 0 to understand what each existing custom eval measures, and (in publish mode) **before any update** — `create_or_update_llmobs_evaluator` is full-replace, so you must round-trip the full config to avoid clobbering fields. Not all evaluators have a stored config (notably `source=ootb`); a not-found error there is expected — skip those. |
| `create_or_update_llmobs_evaluator` | *(publish mode)* Write an LLM-judge evaluator config to Datadog. Full-replace semantics: any omitted optional field resets to its default. See "Publishing Conventions" for required fields and structured output → JSON schema mapping. |
| `delete_llmobs_evaluator` | *(publish mode)* Only used if the user explicitly asks to remove an evaluator. Never invoke speculatively. |

### Key `get_llmobs_span_content` Patterns

Use the `path` parameter to extract targeted data without fetching full payloads:

| Field | Path | What you get |
|-------|------|-------------|
| `messages` | `$.messages[0]` | System prompt (first message, usually `system` role) |
| `messages` | `$.messages[-1]` | Last assistant response |
| `messages` | *(no path)* | Full conversation including tool calls |
| `input` / `output` | — | Span I/O |
| `documents` | — | Retrieved documents (RAG apps) |
| `metadata` | — | Custom metadata (prompt versions, feature flags, user segments) |

### How to Use `search_llmobs_spans`

Additional filters combine with space (AND): `@status:error @ml_app:my-app`. Dedicated params (`span_kind`, `root_spans_only`, `ml_app`) work alongside `query`, but `query` takes precedence over `tags`.

To find spans with a specific eval: `@evaluations.custom.<eval_name>:*` — you can only query for eval *presence*, not specific results.

To detect whether the app uses **sessions**: `session_id:*` matches any span carrying a `session_id` (`session_id` is a first-class field — no `@` prefix). The Phase 1 session probe uses this to gate session-scope evaluators.

### Parallelization Rules

1. **`get_llmobs_span_details`**: Group span_ids by trace_id. One call per trace_id with ALL its span_ids. Issue ALL calls for a page in a **single message**.
2. **`get_llmobs_span_content`**: Each call is independent — always issue ALL in a single message.
3. **`get_llmobs_trace` / `get_llmobs_agent_loop`**: Parallelize across different traces in a single message.
4. **Pipeline parallelism**: Start `get_llmobs_span_details` for page 1 results immediately — don't wait to collect all pages.

---

## Evaluator SDK Reference

> **Applies to `sdk_code` mode only.** In `data_only` mode, use this section as domain context when writing rubric prompts — no SDK classes are emitted.

### Imports

```python
# Core classes
from ddtrace.llmobs._experiment import BaseEvaluator, EvaluatorContext, EvaluatorResult

# LLM-as-judge
from ddtrace.llmobs._evaluators.llm_judge import (
    LLMJudge,
    BooleanStructuredOutput,
    ScoreStructuredOutput,
    CategoricalStructuredOutput,
)

# Built-in evaluators (use only if needed)
from ddtrace.llmobs._evaluators.format import JSONEvaluator, LengthEvaluator
from ddtrace.llmobs._evaluators.string_matching import StringCheckEvaluator, RegexMatchEvaluator
```

Only import what the generated file actually uses.

### EvaluatorContext (what `evaluate()` receives)

```python
@dataclass(frozen=True)
class EvaluatorContext:
    input_data: dict[str, Any]          # Task inputs (from dataset record, NOT from span)
    output_data: Any                     # Task output (from task function return, NOT from span)
    expected_output: Optional[JSONType] = None  # Ground truth (if available)
    metadata: dict[str, Any] = {}        # Additional metadata
    span_id: Optional[str] = None        # LLMObs span ID
    trace_id: Optional[str] = None       # LLMObs trace ID
```

**Important — span data vs evaluator data**: When exploring production traces, you see span I/O (e.g., `input.value`, `output.messages`). But evaluators run in offline experiments where `input_data` and `output_data` come from the user's **dataset records and task function**, not from spans. The dataset schema is user-defined and may not match span structure. Write evaluator prompts with generic `{{input_data}}` / `{{output_data}}` placeholders and add comments describing what data the evaluator was designed for, so the user can adapt to their dataset shape.

### EvaluatorResult (what `evaluate()` returns)

```python
EvaluatorResult(
    value=...,                    # Required. JSONType (str, int, float, bool, None, list, dict)
    reasoning="...",              # Optional. Explanation string
    assessment="pass" or "fail",  # Optional. Pass/fail assessment
    metadata={...},              # Optional. Evaluation metadata dict
    tags={...},                  # Optional. Tags dict
)
```

### LLMJudge — LLM-as-Judge Evaluator

```python
judge = LLMJudge(
    user_prompt="...",              # Required. Supports {{template_vars}}
    system_prompt="...",            # Optional. Does NOT support template vars
    structured_output=...,          # Optional. Boolean/Score/Categorical output, or a dict for custom JSON schema
    provider="openai",              # "openai" | "anthropic" | "azure_openai" | "vertexai" | "bedrock"
    model="gpt-4o",                # Model identifier
    model_params={"temperature": 0.0},  # Optional. Passed to LLM API
    name="eval_name",              # Optional. Must match ^[a-zA-Z0-9_-]+$
)
```

**Template variables** in `user_prompt`: `{{input_data}}`, `{{output_data}}`, `{{expected_output}}`, `{{metadata.key}}` — resolved from `EvaluatorContext` fields via dot-path into nested dicts.

### Structured Output Types

**Boolean** — true/false with optional pass/fail:

```python
BooleanStructuredOutput(
    description="Whether the response is factually accurate",
    reasoning=True,                    # Include reasoning field in LLM response
    reasoning_description=None,        # Optional custom description for reasoning field
    pass_when=True,                    # True → pass when true, False → pass when false, None → no assessment
)
```

**Score** — numeric within a range with optional thresholds:

```python
ScoreStructuredOutput(
    description="Helpfulness score",
    min_score=1,                       # Minimum possible score
    max_score=10,                      # Maximum possible score
    reasoning=True,
    reasoning_description=None,
    min_threshold=7,                   # Scores >= 7 pass (optional)
    max_threshold=None,                # Scores <= N pass (optional)
)
```

**Categorical** — select from predefined categories:

```python
CategoricalStructuredOutput(
    categories={
        "correct": "The response correctly answers the question",
        "partially_correct": "The response is partially correct but missing key information",
        "incorrect": "The response is factually wrong or irrelevant",
    },
    reasoning=True,
    reasoning_description=None,
    pass_values=["correct"],           # Which categories count as passing (optional)
)
```

**Custom JSON schema** — arbitrary structured responses for multi-dimensional evals:

```python
# Pass a raw dict as structured_output — used as the JSON schema directly
structured_output={
    "type": "object",
    "properties": {
        "relevance": {"type": "boolean", "description": "Whether the response addresses the question"},
        "confidence": {"type": "number", "description": "Confidence score (0.0 to 1.0)"},
        "reasoning": {"type": "string", "description": "Explanation for the evaluation"},
    },
    "required": ["relevance", "confidence", "reasoning"],
    "additionalProperties": False,
}
```

Always write standard JSON schema — the SDK adapts it per provider automatically (e.g., Anthropic doesn't support `minimum`/`maximum` on number fields, so the SDK moves range constraints into the `description`; Vertex AI converts `const`/`anyOf` to `enum`). The full parsed JSON dict becomes the eval `value`; a `"reasoning"` key (if present) is automatically extracted. No automatic pass/fail assessment.

### LLMJudge Prompt Guidelines

The `structured_output` parameter enforces the response format via JSON schema. **Do not** prescribe the format in the prompt (no "Answer YES/NO", "Rate 1-10", etc.). Instead, describe the **evaluation criteria** and let the structured output handle the format.

- **system_prompt**: Set the judge's role and the app's domain context. Does NOT support template vars.
- **user_prompt**: Present the data via `{{input_data}}` / `{{output_data}}`, then describe what good vs. bad looks like for this dimension.

### BaseEvaluator — Custom Code-Based Evaluator

For deterministic checks that do not need LLM judgment:

```python
class MyEvaluator(BaseEvaluator):
    def __init__(self, name=None, ...custom_params...):
        super().__init__(name=name)
        self._param = ...  # Store config as private attrs

    def evaluate(self, context: EvaluatorContext) -> EvaluatorResult:
        # Access: context.input_data, context.output_data, context.expected_output, context.metadata
        # Must NOT modify self attributes (thread safety)
        passed = ...  # Your logic here
        return EvaluatorResult(
            value=passed,
            reasoning="...",
            assessment="pass" if passed else "fail",
        )
```

### Built-in Evaluators

```python
# Validate JSON syntax + optional required keys
JSONEvaluator(required_keys=["name", "age"], output_extractor=None, name=None)

# Validate length (characters, words, or lines)
LengthEvaluator(count_by="words", min_length=10, max_length=500, output_extractor=None, name=None)
# count_by: "characters" | "words" | "lines"

# String matching
StringCheckEvaluator(operation="contains", expected="success", case_sensitive=False, name=None)
# operation: "eq" | "ne" | "contains" | "icontains"

# Regex matching
RegexMatchEvaluator(pattern=r"\d{4}-\d{2}-\d{2}", match_mode="search", name=None)
# match_mode: "search" | "match" | "fullmatch"
```

### Evaluator Type Decision Matrix

| Signal | Evaluator Type |
|--------|---------------|
| Output must be valid JSON | `JSONEvaluator` |
| Output must match a regex pattern | `RegexMatchEvaluator` |
| Output has length constraints | `LengthEvaluator` |
| Output must contain/not contain specific strings | `StringCheckEvaluator` |
| Semantic quality judgment (tone, accuracy, completeness) | `LLMJudge` + `BooleanStructuredOutput` |
| Graded quality on a scale | `LLMJudge` + `ScoreStructuredOutput` |
| Classification into categories | `LLMJudge` + `CategoricalStructuredOutput` |
| Multi-dimensional judgment (evaluate several aspects at once) | `LLMJudge` + custom JSON schema `dict` |
| Complex domain logic combining multiple checks | `BaseEvaluator` subclass |

### Source Verification

If you have access to dd-trace-py locally, verify the API surface by reading the corresponding modules:

- `ddtrace.llmobs._evaluators.llm_judge` — `LLMJudge`, `BooleanStructuredOutput`, `ScoreStructuredOutput`, `CategoricalStructuredOutput`
- `ddtrace.llmobs._experiment` — `BaseEvaluator`, `EvaluatorContext`, `EvaluatorResult`
- `ddtrace.llmobs._evaluators.format` — `JSONEvaluator`, `LengthEvaluator`
- `ddtrace.llmobs._evaluators.string_matching` — `StringCheckEvaluator`, `RegexMatchEvaluator`

---

## Workflow

### Phase 0: Resolve Inputs & Entry Mode

**Entry mode detection:**

| Mode | Signal | Behavior |
|------|--------|----------|
| **Cold Start** | Only `ml_app` provided (no RCA, no hypothesis) | Full open discovery — understand what the app does, identify quality dimensions worth measuring, propose evals for coverage |
| **From RCA** | Conversation contains an RCA report or user provides a failure hypothesis | Skip open discovery — use existing failure taxonomy as eval targets |

**Parse arguments**: Extract `ml_app` (first non-flag argument), `--timeframe` (default `now-7d`), `--trace-limit` (default `20`), `--sdk-code`, `--data-only`, and `--emit-dataset <path>` flags. Set `output_mode` as follows (at most one of the three mode flags may be set; error if more than one is present):

- `--emit-dataset <path>` set → `output_mode = emit_dataset`. Skip the rest of the workflow entry-mode logic and jump directly to **Phase 3D** below.
- `--sdk-code` set → `output_mode = sdk_code`.
- `--data-only` set → `output_mode = data_only`.
- otherwise → `output_mode = publish` (the default — propose online evaluators, gated on user confirmation, created as disabled drafts).

**Resolution steps:**

1. If `ml_app` not provided → ask the user.
2. Auto-detect entry mode:
   - If the conversation contains an RCA report (look for "Failure Taxonomy" heading, structured failure modes, or severity ratings) → `from_rca`. Extract the taxonomy.
   - If the user provides a free-text failure hypothesis (e.g., "the system prompt lacks grounding") → `from_rca`. Use the hypothesis as the starting eval target.
   - Otherwise → `cold_start`.
3. If `timeframe` not provided → default to `now-7d`.
4. **Map existing eval coverage** — **skip if `output_mode = data_only`** (there is no Datadog eval project to check coverage against): Call `list_llmobs_evals` (org-wide; filter the result client-side to entries where `ml_app == <ml_app>`). Then, for each eval with `source=custom`, call `get_llmobs_evaluator(eval_name=...)` to inspect its prompt template, target, sampling, and filter, and infer which quality dimension it covers. Issue all evaluator calls in a **single message** (parallelize). Skip `source=ootb` evals — their names are self-describing and they may not have a fetchable config.

   By the end of this step you have a complete coverage map: `{eval_name → source, enabled, dimension}`. Carry this into Phase 2 for deduplication.

   **In `publish` mode, also note any template-variable convention** the existing custom evaluators already use (so a new suite reads consistently). Online evaluator templates resolve against the **full span JSON**, not against `EvaluatorContext`. See the "Online Template Variables" section under "Publishing Conventions" for the supported syntax (`{{span_input}}`, `{{span_output}}`, dot-paths, array selectors, filter accessors).

5. **Notebook context detection**: Scan the current conversation for a Datadog notebook URL that was produced by `/eval-trace-rca` (pattern: `https://app.datadoghq.com/notebook/{numeric-id}`). If found, store it as `rca_notebook_url` and extract the numeric ID as `rca_notebook_id`. This is used after Phase 3 to offer appending the evaluator suite to that notebook instead of creating a new one.

---

### Phase 1: Explore Traces & Identify Eval Targets

**Goal**: Sample production traces, understand what the app does, and identify quality dimensions worth measuring.

#### Cold Start Path

1. **Sample the app**: `search_llmobs_spans(query="@ml_app:\"<ml_app>\" @status:ok", root_spans_only=true, limit=50, from=<timeframe>)`. Filter by `@status:ok` — error spans have no output to evaluate.

   **Session probe** *(gates session-scope proposals; `publish` mode)*: in the same message, also call `search_llmobs_spans(query="@ml_app:\"<ml_app>\" session_id:*", limit=20, from=<timeframe>)`.
   - **≥ 1 result** → set `sessions_present = true`. Note the distinct `session_id` values and, critically, whether the same `session_id` appears across **multiple `trace_id`s** — that cross-trace span is the real signal that a session carries context worth a session-scope evaluator. (A `session_id` that only ever maps to one trace adds nothing over trace scope.)
   - **0 results** → `sessions_present = false`. Do **not** propose any session-scope evaluator; record a one-line "session scope skipped — no `session_id` on sampled spans" note for the proposal.

2. **Profile the app and identify evaluation target spans**: Call `get_llmobs_span_details` for span_ids grouped by trace_id. Inspect `content_info` to classify:

   | Signal | App Profile |
   |--------|------------|
   | `content_info` has `messages` | LLM/chat app |
   | `content_info` has `documents` | RAG app |
   | Spans include `agent` kind | Agent app |
   | `content_info` has `metadata` | Has custom metadata |
   | Multiple span kinds in one trace (`agent` + `tool` / `retrieval` + `llm` from `get_llmobs_trace`) | Multi-step app — at least one trace-scope evaluator likely belongs in the suite (`publish` mode) |
   | Same `session_id` across **multiple `trace_id`s** (from the session probe) | Multi-trace sessions — at least one session-scope evaluator likely belongs in the suite (`publish` mode, gated on `sessions_present`) |

   For agent/multi-step apps, also call `get_llmobs_trace` on 2-3 traces to see the full span hierarchy. Compare `content_info` between the root span and its sub-spans. Then ask **two** questions for each candidate quality dimension, in this order:

   1. **Does the verdict depend on more than one span?** (e.g., faithfulness depends on a `retrieval` span's documents AND an `llm` span's answer; goal completion depends on the chain of `tool` calls AND the final response.) If yes → **trace scope** in `publish` mode. Don't try to compress this into a single span.
   2. **Only if the answer to (1) is no**: pick the single span with the richest signal for that dimension (root has the summary; LLM sub-spans have the full system prompt + tool call results + reasoning chain).

   Record the span-kind histogram (agent + tool + llm + retrieval) — multiple kinds under one root is a strong signal you'll have at least one trace-scope evaluator in the suite. See Phase 2's "Span vs. Trace vs. Session Scope Classification" for the mandatory walk-through of canonical trace-scope use cases (and, when `sessions_present`, the canonical session-scope use cases).

3. **Extract content and identify targets**: Call `get_llmobs_span_content` for representative spans. Fetch fields based on app profile:

   | App Profile | Fields to Fetch |
   |------------|----------------|
   | LLM/chat | `messages` (`path=$.messages[0]` for system prompt), `output` |
   | RAG | `documents`, `input`, `output` |
   | Agent | `get_llmobs_agent_loop` for the agent span, then `messages` for detail |
   | Any with metadata | `metadata` |

   Issue all calls in a single message. As you read, capture two streams of signal:

   **Generic quality signals** — what does "success" look like? What variance exists across outputs? Each observed quality dimension becomes a candidate evaluator, with the traces you've just read as evidence. Also look for safety signals (scope violations, sensitive data in outputs, out-of-character responses) and add a safety evaluator if you find them.

   **Domain signals** — these become the *domain-specific evaluator* category in Phase 2 (the highest-leverage category). For every 5–10 traces, write down:
   - **Recurring intents / question categories** — what classes of request does this app handle? (`applying for benefit X`, `comparing flight options`, `summarizing a policy`, `creating a widget`)
   - **Entities the app emits in outputs** — URLs, agency / company names, code identifiers, monetary amounts, dates, IDs, file paths, phone numbers. Note which ones the user *acts* on downstream (those are worth a correctness evaluator) versus which are passing references.
   - **Tool argument shapes** (for agent apps) — name each tool the agent calls and the rough schema of its inputs. Tools with non-trivial schemas (≥ 3 fields, structured types) are candidates for argument-correctness evaluators.
   - **Persona / voice rules** — does the app always cite a source, always refuse certain topics (medical, legal, financial advice), always speak in a particular tone? Extract the rules implicitly followed across observed outputs.
   - **Failure modes specific to the domain** — fabricated identifiers, outdated policy references, currency / locale mismatches, off-by-one errors in IDs, wrong units. One observed instance is enough to seed a candidate evaluator.

   Don't try to enumerate domain signals exhaustively before reading traces — let the patterns surface as you read. The goal is breadth in the eventual proposal, not completeness in this exploration step.

#### From RCA Path

1. Extract the failure taxonomy from the RCA report. Each failure mode with High or Medium severity becomes an eval target. Also run the Phase 1 **session probe** (`query="session_id:*"`) to set `sessions_present` — a failure that only manifests across a multi-trace conversation (lost context, repeated mistakes, mounting frustration) is a session-scope target.

2. **Check root cause categories for infrastructure failures.** Before proposing evaluators, scan the Root Cause column of the taxonomy for any of: `Instrumentation Deficiency`, `Harness Deficiency`, `Runtime Error`, `Upstream Data Issue`, or any other root cause that points to infrastructure/environment rather than model behavior. If any are present, pause and ask:

   > "Some failure modes were diagnosed as infrastructure or instrumentation issues rather than model behavior (e.g., `{list the infra root causes}`). Evaluators can be designed two ways:
   > - **Behavior-targeted** (recommended for ongoing quality): measure whether the model produces correct, specific output — useful once the infrastructure is fixed and you want to track real quality
   > - **Artifact-targeted** (useful as regression guard): detect the specific broken output observed (e.g., generic placeholder responses) — catches regressions if the infrastructur...