Skip to main content
skill CX Codex

langsmith-trace-analyzer — Claude Code skill for trace analysis

The langsmith-trace-analyzer Claude Code skill downloads and analyzes LangSmith traces. It helps investigate agent runs and compare evaluation results.

MIT ★ 106 updated 2026-11-11

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-code
Open on GitHub

What langsmith-trace-analyzer does

The skill exports LangSmith traces as JSON, groups them by outcome, and produces debugging reports. Runs can be selected by project, filter, or time range. Analysis compares passed and failed runs, summarizes messages, tool calls, tokens, and duration, and flags repeated tool use and quick failures.

The package includes Python scripts for downloading and analysis, plus a TypeScript downloader. It requires the langsmith and langsmith-fetch packages and a LANGSMITH_API_KEY. Trace payloads can include user prompts and other sensitive runtime data, so unsanitized exports should not be committed to a repository.

What it helps with

  • find traces by project, metadata, or time range
  • export LangSmith runs as JSON
  • sort runs into passed, failed, and error groups
  • compare metrics across passed and failed runs
  • spot repeated tool calls and quick failures

Example prompts for the agent

  • Find traces for my-project from the last 24 hours and save them to ./traces.
  • Compare passed and failed runs in ./traces, then write the report to comparison.md.
  • Analyze the traces in ./traces and return a JSON summary.

How to install langsmith-trace-analyzer

1Skills CLI (Vercel)

npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-code

2Manually: put the files in place

~/.claude/skills/langsmith-trace-analyzer/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/langsmith-trace-analyzer/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about langsmith-trace-analyzer

How do I install langsmith-trace-analyzer in Claude Code?

Run npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-code. The skill file is placed at ~/.claude/skills/langsmith-trace-analyzer/SKILL.md.

What does langsmith-trace-analyzer do?

It downloads and groups LangSmith traces, then reports patterns and compares passed and failed runs. The analysis covers messages, tool calls, tokens, and duration.

What does langsmith-trace-analyzer require?

Install langsmith and langsmith-fetch, and set LANGSMITH_API_KEY. The package installation command is uv pip install langsmith langsmith-fetch.

Source text

SKILL.md · MIT
---
name: langsmith-trace-analyzer
description: "Fetch, organize, and analyze LangSmith traces for debugging and evaluation. Use when you need to: query traces/runs by project, metadata, status, or time window; download traces to JSON; organize outcomes into passed/failed/error buckets; analyze token/message/tool-call patterns; compare passed vs failed behavior; or investigate benchmark and production failures."
---

# LangSmith Trace Analyzer

Use this skill to move from raw LangSmith traces to actionable debugging/evaluation insights.

## Quick Start

```bash
# Install dependencies
uv pip install langsmith langsmith-fetch

# Auth
export LANGSMITH_API_KEY=<your_langsmith_api_key>
```

### Fast workflow

1. Download traces with `scripts/download_traces.py` (or `scripts/download_traces.ts`).
2. Analyze downloaded JSON with `scripts/analyze_traces.py`.
3. Load targeted references only when needed:
   - `references/filtering-querying.md` for query/filter syntax
   - `references/analysis-patterns.md` for deeper diagnostics
   - `references/benchmark-analysis.md` for benchmark-specific workflows

## Decision Guide

1. **Known trace IDs**  
Use `langsmith-fetch trace <id>` directly, or `--trace-ids` in downloader scripts.

2. **Need to discover traces first**  
Use LangSmith SDK `list_runs/listRuns` with filters, then download selected trace IDs.

3. **Need aggregate insights**  
Run `analyze_traces.py` for summary stats, patterns, and passed-vs-failed comparisons.

## Core Workflows

### 1) Download and organize traces

Python:

```bash
uv run skills/langsmith-trace-analyzer/scripts/download_traces.py \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces \
  --organize
```

TypeScript:

```bash
ts-node skills/langsmith-trace-analyzer/scripts/download_traces.ts \
  --project "my-project" \
  --filter "job_id=abc123" \
  --last-hours 24 \
  --limit 100 \
  --output ./traces
```

Output layout:

```text
traces/
├── manifest.json
└── by-outcome/
    ├── passed/
    ├── failed/
    └── error/
        ├── GraphRecursionError/
        ├── TimeoutError/
        └── DaytonaError/
```

Notes:
- Python script supports `--organize/--no-organize`.
- Both scripts use SDK filtering plus `langsmith-fetch` for full trace payload export.

### 2) Analyze downloaded traces

```bash
# Markdown report
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --output report.md

# JSON output
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --json

# Compare passed vs failed (expects by-outcome folders)
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --compare --output comparison.md
```

The analyzer reports:
- message/tool-call/token/duration summaries
- top tool usage
- anomaly patterns (high message count, repeated tools, quick failures)
- passed-vs-failed metric deltas when comparison is enabled

### 3) Query traces correctly (SDK)

Use official LangSmith run filter syntax via `filter` and/or `start_time`:

```python
from datetime import datetime, timedelta, timezone
from langsmith import Client

client = Client()

start = datetime.now(timezone.utc) - timedelta(hours=24)
filter_query = 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))'

runs = client.list_runs(
    project_name="my-project",
    is_root=True,
    start_time=start,
    filter=filter_query,
)
```

For TypeScript:

```ts
import { Client } from "langsmith";

const client = new Client();
for await (const run of client.listRuns({
  projectName: "my-project",
  isRoot: true,
  filter: 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))',
})) {
  console.log(run.id, run.status);
}
```

## Accuracy and Schema Notes

- LangSmith run fields are commonly top-level (`status`, `error`, `total_tokens`, `start_time`, `end_time`).
- Some exported traces also include nested metadata (`metadata` or `extra.metadata`) and/or `messages`.
- `analyze_traces.py` is resilient to multiple payload shapes, including raw array payloads.
- For full conversation content, prefer downloaded trace payloads over bare `list_runs` results.

## Troubleshooting

| Issue | Likely Cause | Action |
|---|---|---|
| `LANGSMITH_API_KEY` missing | Auth not configured | `export LANGSMITH_API_KEY=<your_langsmith_api_key>` |
| No runs returned | Wrong project/filter/time range | Verify project name and filter syntax |
| Empty/partial message arrays | Run schema differs or incomplete data | Use downloaded trace JSON and inspect `status/error` fields |
| JSON parse error on downloaded files | Bad/incomplete export | Re-download trace; use `--format raw` paths in scripts |
| Re-downloading same traces repeatedly | Existing files in nested folders | Use current scripts (they check existing files across output tree) |

## Safety for Open Source

- Do not commit downloaded trace artifacts (`manifest.json`, trace JSON dumps) unless sanitized.
- Trace payloads can contain user prompts, outputs, metadata, and other sensitive runtime data.
- Keep this skill repository focused on scripts/templates, not production trace exports.

## Resources

### scripts/

- `scripts/download_traces.py`: Python downloader + organizer
- `scripts/download_traces.ts`: TypeScript downloader + organizer
- `scripts/analyze_traces.py`: Offline analysis and reporting

### references/

- `references/filtering-querying.md`: LangSmith query/filter examples
- `references/analysis-patterns.md`: Diagnostic patterns and heuristics
- `references/benchmark-analysis.md`: Benchmark-oriented analysis

© soba-labs · MIT · soba-labs/langchain-agent-skills

Similar skills

All skills for AI and ML →

From the same set9

langsmith-deployment This Claude Code skill manages agent server deployments in LangSmith Deployment. It comes into play when validating langgraph.json, changing revisions, or diagnosing runtime issues. CX Codex +1 ★ 106repo stars langgraph-testing-evaluation langgraph-testing-evaluation — a skill for Claude Code tests and evaluates LangGraph and LangChain agents. It supports test scaffolds, trajectory scoring, dataset runs, and version comparisons. CX Codex +1 ★ 106repo stars langgraph-state-management The langgraph-state-management Claude Code skill helps shape state schemas for LangGraph applications and diagnose update behavior. It applies to reducer design, persistence setup, and checkpoint debugging. CX Codex +1 ★ 106repo stars langgraph-project-setup The langgraph-project-setup Claude Code skill scaffolds and configures LangGraph projects. It activates for project structure, langgraph.json, provider variables, dependency setup, and configuration troubleshooting. CX Codex +1 ★ 106repo stars langgraph-error-handling The langgraph-error-handling Claude Code skill adds error-handling patterns to LangGraph. It applies to transient failures, model recovery, and workflows that need human approval. CX Codex +1 ★ 106repo stars langgraph-agent-patterns The langgraph-agent-patterns Claude Code skill helps build agent coordination patterns for LangGraph applications. It activates when choosing a pattern or debugging how agents work together. CX Codex +1 ★ 106repo stars deepagents-setup-configuration The deepagents-setup-configuration Claude Code skill scaffolds and checks Deep Agents projects in Python or JavaScript. It helps configure subagents, persistence, and human approval. CX Codex +1 ★ 106repo stars deepagents-planning-todos The deepagents-planning-todos Claude Code skill shows how to plan Deep Agents work with write_todos. It covers task breakdown, progress tracking, and diagnosing todos that remain unfinished. CX Codex +1 ★ 106repo stars

Project sponsors

Companies that keep this analytics open