Skip to main content
skill CX Codex

mlflow-trace-analysis — Claude Code skill: analysis and install

The mlflow-trace-analysis Claude Code skill examines MLflow traces to explain agent failures, errors, and quality issues. It handles single traces and multi-turn sessions.

Apache-2.0 178 installs ★ 91 updated 2026-10-23

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add mlflow/skills --skill mlflow-trace-analysis -a claude-code
Open on GitHub

What mlflow-trace-analysis does

The skill traces observed behavior back to span inputs and outputs in MLflow data. It investigates errors, unexpected responses, quality concerns, latency, and failures that carry across conversation turns. It does not edit application code.

Analysis needs a trace or session ID and a configured backend. The skill fetches full JSON with mlflow traces get, then inspects the span tree, feedback, and error events against the expected behavior. For an unknown trace ID, use mlflow-trace-retrieval or consult the search guide and installed CLI help.

The findings distinguish observed evidence from plausible causes and unresolved questions; a hypothesis is not reported as a confirmed root cause. Reproduction is limited to one attempt through the usual instrumented entry point, and only when safe.

What it helps with

  • locate the first span where an agent response diverged from expectations
  • explain an error using span status and exception events
  • trace how one failed turn affects later responses
  • separate application failures from feedback-scoring errors
  • inspect latency across nested and overlapping spans

Example prompts for the agent

  • Analyze trace TRACE_ID and find where the agent response diverged from expectations.
  • Follow session SESSION_ID to see how a failure in one turn affected later replies.
  • Review feedback and spans in trace TRACE_ID; separate evidence from possible causes.

How to install mlflow-trace-analysis

1Skills CLI (Vercel)

npx skills add mlflow/skills --skill mlflow-trace-analysis -a claude-code

2Manually: put the files in place

~/.claude/skills/mlflow-trace-analysis/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/mlflow-trace-analysis/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about mlflow-trace-analysis

How do I install mlflow-trace-analysis in Claude Code?

Run npx skills add mlflow/skills --skill mlflow-trace-analysis -a claude-code. The skill file is installed at ~/.claude/skills/mlflow-trace-analysis/SKILL.md.

What does mlflow-trace-analysis do?

It examines MLflow traces and multi-turn sessions, connecting span inputs and outputs with errors, unexpected responses, quality issues, and latency. It provides diagnostic findings without changing application code.

What does mlflow-trace-analysis need to run?

It needs a trace or session ID and a configured backend. To find an unknown trace ID, use mlflow-trace-retrieval or the search guide and installed MLflow CLI help.

Source text

SKILL.md · Apache-2.0
---
name: mlflow-trace-analysis
description: Analyze MLflow traces, agent failures, and multi-turn chat sessions. Explain errors, unexpected output, quality issues, latency, or cross-turn propagation without editing application code.
---

# Analyze MLflow traces

Use the trace ID or session ID, configured backend, reported behavior and expected outcome. Reuse traces already fetched. For a multi-turn conversation, read [session analysis](references/session-analysis.md).

## Prerequisites

Before searching for an unknown trace ID, read [trace retrieval](../mlflow-trace-retrieval/SKILL.md) for filters, time bounds and pagination. Reuse an already fetched trace. If that skill is not installed, use the [search guide](https://mlflow.org/docs/latest/genai/tracing/search-traces.md) and installed CLI help.

Prefer existing evidence. Only when no usable trace exists, reproduce once through the normal instrumented entry point when safe; otherwise report the missing evidence. Record HUMAN feedback only when authorized and attached to an identified trace.

## Step 1: Fetch the trace

Fetch full JSON once to a file; large piped CLI output can be truncated by the execution tool. `get` needs no `--output` flag. Avoid `--extract-fields` for nested span diagnosis.

```bash
mlflow traces get --trace-id <TRACE_ID> > /tmp/trace.json
```

## Step 2: Inspect the span tree

Summarize state, assessments and parent links:

```bash
jq '{state: .info.state,
     spans: [.data.spans[] | {span_id, parent_span_id, name, status}],
     error_spans: [.data.spans[] | select(.status.code == "STATUS_CODE_ERROR") | .name],
     assessments: .info.assessments}' /tmp/trace.json
```

## Step 3: Read feedback

Read feedback **source, value, top-level rationale, validity and scope** before forming a hypothesis. HUMAN feedback describes the reported experience; corroborate it with spans. LLM_JUDGE/CODE feedback may be wrong or misconfigured. `feedback.error` means scoring failed, not that the application failed; exclude errored/overridden judgments from quality conclusions. Genuine zero/false values remain valid. Expectations are desired-output labels, not scores.

## Step 4: Follow inputs and outputs

Follow actual inputs and outputs from root to the first divergence, then downstream effects. Raw MLflow span I/O is usually serialized JSON in `attributes["mlflow.spanInputs"]` / `attributes["mlflow.spanOutputs"]`; parse it before interpreting it. If the root is an empty wrapper, inspect descendants. For other OTel sources or versions, inspect raw keys instead of assuming this schema.

## Step 5: Check errors and performance

Check exception events and status messages. `OK` means execution success, not correct output; `STATUS_CODE_UNSET` is not itself an error. For performance, distinguish enclosing wrappers, overlapping child work and uninstrumented gaps; repeated names alone do not prove retries. Inspect recorded token usage rather than inventing cost or assuming a token spike proves injection.

## Step 6: Check the source

Correlate the decisive span's observed data with relevant source/configuration when available. Function names are clues; autologged names follow framework conventions. Separate observed facts, plausible causes and missing evidence. Do not edit code, create scorers or run an evaluation for diagnosis alone.

See the [refund-policy evidence table](references/session-analysis.md#worked-example-refund-policy-propagation) for a retrieval miss, unsupported answer and later propagation.

## Verification

Report observed versus expected behavior, trace and decisive span IDs, the evidence chain, confidence and the smallest likely remedy. Confirm claims against actual span I/O and feedback semantics, not previews alone. If evidence is missing, say what would resolve the uncertainty; do not promote a hypothesis to a confirmed root cause.

## References

- Fields, units, CLI versus SDK representation, OTel attributes or assessment interpretation: [trace structure](references/trace-structure.md).
- Bounded batches, recurring failures or component bottlenecks: [cross-trace analysis](references/cross-trace-analysis.md).
- Multi-turn propagation: [session analysis](references/session-analysis.md).
- A user-requested implementation: pass diagnosis and trace ID to `mlflow-agent-improvement` when installed; investigation itself stops at evidence and recommendations.

© mlflow · Apache-2.0 · mlflow/skills

Similar skills

All skills for AI and ML →

From the same set17

The whole set →
mlflow-agent Routes MLflow requests to the right child skill for tracing, evaluation, debugging, or documentation. Use it when the task has no specific MLflow workflow selected. CX Codex +1 882installs sagemaker-mlflow sagemaker-mlflow connects the agent to SageMaker Managed MLflow via resource ARN (mlflow-app or mlflow-tracking-server), then hands off to other MLflow skills through MLFLOW_TRACKING_URI. CX Codex +1 394installs fix-agent-issue fix-agent-issue — a skill that walks the agent through EXPLORE → PLAN → IMPLEMENT → VERIFY so a behavior change rests on a trace and a test, not a guess. CX Codex +1 376installs build-a-scorer build-a-scorer helps design and iterate MLflow judges and scorers for LLM apps or RAG agents. It defines atomic criteria and picks the cheapest reliable check. CX Codex +1 339installs mlflow-evaluation The mlflow-evaluation Claude Code skill configures MLflow scorers and evaluates an agent, dataset, precomputed outputs, or traces. It checks GenAI quality and inspects evaluation results. CX Codex +1 184installs mlflow-trace-metrics The mlflow-trace-metrics Claude Code skill queries aggregate trace, span, and assessment metrics from MLflow. Runs telemetry summaries for latency, token counts, and evaluation scores. CX Codex +1 174installs mlflow-trace-retrieval The mlflow-trace-retrieval Claude Code skill retrieves MLflow traces by ID or searches them with filters. It runs for trace lookups based on experiments, runs, users, time windows, and other fields. CX Codex +1 174installs mlflow Equips Claude Code with workflows for tracing, debugging, and evaluating GenAI apps using MLflow. Helpful when instrumenting LLMOps. ★ 91repo stars

Project sponsors

Companies that keep this analytics open