langsmith-trace-analyzer — Claude Code skill for trace analysis
The langsmith-trace-analyzer Claude Code skill downloads and analyzes LangSmith traces. It helps investigate agent runs and compare evaluation results.
MIT ★ 106 updated 2026-11-11
npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-codeWhat langsmith-trace-analyzer does
The skill exports LangSmith traces as JSON, groups them by outcome, and produces debugging reports. Runs can be selected by project, filter, or time range. Analysis compares passed and failed runs, summarizes messages, tool calls, tokens, and duration, and flags repeated tool use and quick failures.
The package includes Python scripts for downloading and analysis, plus a TypeScript downloader. It requires the langsmith and langsmith-fetch packages and a LANGSMITH_API_KEY. Trace payloads can include user prompts and other sensitive runtime data, so unsanitized exports should not be committed to a repository.
What it helps with
- find traces by project, metadata, or time range
- export LangSmith runs as JSON
- sort runs into passed, failed, and error groups
- compare metrics across passed and failed runs
- spot repeated tool calls and quick failures
Example prompts for the agent
- Find traces for
my-projectfrom the last 24 hours and save them to./traces. - Compare passed and failed runs in
./traces, then write the report tocomparison.md. - Analyze the traces in
./tracesand return a JSON summary.
How to install langsmith-trace-analyzer
1Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-code2Manually: put the files in place
~/.claude/skills/langsmith-trace-analyzer/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/langsmith-trace-analyzer/
1Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a codex2Manually: put the files in place
.agents/skills/langsmith-trace-analyzer/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a cursor2Manually: put the files in place
.cursor/skills/langsmith-trace-analyzer/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install soba-labs/langchain-agent-skills skills/langsmith-trace-analyzer2Manually: put the files in place
.github/skills/langsmith-trace-analyzer/SKILL.md1Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a gemini-cli2Platform CLI
gemini skills install https://github.com/soba-labs/langchain-agent-skills --path skills/langsmith-trace-analyzerManually: put the files in place
.devin/skills/langsmith-trace-analyzer/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about langsmith-trace-analyzer
How do I install langsmith-trace-analyzer in Claude Code?
Run npx skills add soba-labs/langchain-agent-skills --skill langsmith-trace-analyzer -a claude-code. The skill file is placed at ~/.claude/skills/langsmith-trace-analyzer/SKILL.md.
What does langsmith-trace-analyzer do?
It downloads and groups LangSmith traces, then reports patterns and compares passed and failed runs. The analysis covers messages, tool calls, tokens, and duration.
What does langsmith-trace-analyzer require?
Install langsmith and langsmith-fetch, and set LANGSMITH_API_KEY. The package installation command is uv pip install langsmith langsmith-fetch.
Source text
SKILL.md · MIT
---
name: langsmith-trace-analyzer
description: "Fetch, organize, and analyze LangSmith traces for debugging and evaluation. Use when you need to: query traces/runs by project, metadata, status, or time window; download traces to JSON; organize outcomes into passed/failed/error buckets; analyze token/message/tool-call patterns; compare passed vs failed behavior; or investigate benchmark and production failures."
---
# LangSmith Trace Analyzer
Use this skill to move from raw LangSmith traces to actionable debugging/evaluation insights.
## Quick Start
```bash
# Install dependencies
uv pip install langsmith langsmith-fetch
# Auth
export LANGSMITH_API_KEY=<your_langsmith_api_key>
```
### Fast workflow
1. Download traces with `scripts/download_traces.py` (or `scripts/download_traces.ts`).
2. Analyze downloaded JSON with `scripts/analyze_traces.py`.
3. Load targeted references only when needed:
- `references/filtering-querying.md` for query/filter syntax
- `references/analysis-patterns.md` for deeper diagnostics
- `references/benchmark-analysis.md` for benchmark-specific workflows
## Decision Guide
1. **Known trace IDs**
Use `langsmith-fetch trace <id>` directly, or `--trace-ids` in downloader scripts.
2. **Need to discover traces first**
Use LangSmith SDK `list_runs/listRuns` with filters, then download selected trace IDs.
3. **Need aggregate insights**
Run `analyze_traces.py` for summary stats, patterns, and passed-vs-failed comparisons.
## Core Workflows
### 1) Download and organize traces
Python:
```bash
uv run skills/langsmith-trace-analyzer/scripts/download_traces.py \
--project "my-project" \
--filter "job_id=abc123" \
--last-hours 24 \
--limit 100 \
--output ./traces \
--organize
```
TypeScript:
```bash
ts-node skills/langsmith-trace-analyzer/scripts/download_traces.ts \
--project "my-project" \
--filter "job_id=abc123" \
--last-hours 24 \
--limit 100 \
--output ./traces
```
Output layout:
```text
traces/
├── manifest.json
└── by-outcome/
├── passed/
├── failed/
└── error/
├── GraphRecursionError/
├── TimeoutError/
└── DaytonaError/
```
Notes:
- Python script supports `--organize/--no-organize`.
- Both scripts use SDK filtering plus `langsmith-fetch` for full trace payload export.
### 2) Analyze downloaded traces
```bash
# Markdown report
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --output report.md
# JSON output
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --json
# Compare passed vs failed (expects by-outcome folders)
uv run skills/langsmith-trace-analyzer/scripts/analyze_traces.py ./traces --compare --output comparison.md
```
The analyzer reports:
- message/tool-call/token/duration summaries
- top tool usage
- anomaly patterns (high message count, repeated tools, quick failures)
- passed-vs-failed metric deltas when comparison is enabled
### 3) Query traces correctly (SDK)
Use official LangSmith run filter syntax via `filter` and/or `start_time`:
```python
from datetime import datetime, timedelta, timezone
from langsmith import Client
client = Client()
start = datetime.now(timezone.utc) - timedelta(hours=24)
filter_query = 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))'
runs = client.list_runs(
project_name="my-project",
is_root=True,
start_time=start,
filter=filter_query,
)
```
For TypeScript:
```ts
import { Client } from "langsmith";
const client = new Client();
for await (const run of client.listRuns({
projectName: "my-project",
isRoot: true,
filter: 'and(eq(metadata_key, "job_id"), eq(metadata_value, "abc123"))',
})) {
console.log(run.id, run.status);
}
```
## Accuracy and Schema Notes
- LangSmith run fields are commonly top-level (`status`, `error`, `total_tokens`, `start_time`, `end_time`).
- Some exported traces also include nested metadata (`metadata` or `extra.metadata`) and/or `messages`.
- `analyze_traces.py` is resilient to multiple payload shapes, including raw array payloads.
- For full conversation content, prefer downloaded trace payloads over bare `list_runs` results.
## Troubleshooting
| Issue | Likely Cause | Action |
|---|---|---|
| `LANGSMITH_API_KEY` missing | Auth not configured | `export LANGSMITH_API_KEY=<your_langsmith_api_key>` |
| No runs returned | Wrong project/filter/time range | Verify project name and filter syntax |
| Empty/partial message arrays | Run schema differs or incomplete data | Use downloaded trace JSON and inspect `status/error` fields |
| JSON parse error on downloaded files | Bad/incomplete export | Re-download trace; use `--format raw` paths in scripts |
| Re-downloading same traces repeatedly | Existing files in nested folders | Use current scripts (they check existing files across output tree) |
## Safety for Open Source
- Do not commit downloaded trace artifacts (`manifest.json`, trace JSON dumps) unless sanitized.
- Trace payloads can contain user prompts, outputs, metadata, and other sensitive runtime data.
- Keep this skill repository focused on scripts/templates, not production trace exports.
## Resources
### scripts/
- `scripts/download_traces.py`: Python downloader + organizer
- `scripts/download_traces.ts`: TypeScript downloader + organizer
- `scripts/analyze_traces.py`: Offline analysis and reporting
### references/
- `references/filtering-querying.md`: LangSmith query/filter examples
- `references/analysis-patterns.md`: Diagnostic patterns and heuristics
- `references/benchmark-analysis.md`: Benchmark-oriented analysis
© soba-labs · MIT · soba-labs/langchain-agent-skills