langgraph-testing-evaluation — скилл для Claude Code: тесты агентов
Скилл langgraph-testing-evaluation для Claude Code проверяет и оценивает агентов LangGraph и LangChain. Он подключается к задачам с тестами узлов, оценкой траекторий и сравнением версий агента.
MIT ★ 106 обновлён 2026-11-11
npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation -a claude-codeЧто делает langgraph-testing-evaluation
Скилл помогает писать модульные и интеграционные тесты для агентов LangGraph и LangChain, создавать заготовки тестов и подменять ответы LLM и инструментов. В комплекте есть скрипты для Python и JavaScript/TypeScript, шаблон pytest и пример набора данных.
Для оценки траекторий доступны сопоставление с эталоном и оценка LLM-судьёй. Скрипты также запускают проверки наборов данных через LangSmith и сравнивают две версии агента по качеству и задержке. Для LangSmith-оценки нужен сам сервис LangSmith.
Результаты тестов можно использовать как пороги регрессии в CI, а ошибки разбирать по трассам и экспериментам. Реальные проверки с моделью предлагается держать отдельным небольшим набором, а для повторяемых тестов подменять ввод и вывод LLM и инструментов.
Какие задачи решает
- создать стартовые тесты для графа агента
- проверить маршрутизацию и взаимодействие узлов
- сопоставить траекторию агента с эталоном
- оценить набор примеров в LangSmith
- сравнить качество и задержку двух версий агента
Примеры запросов агенту
- Создай стартовые тесты для графа
my_agent:graphс помощью pytest. - Проверь траекторию
my_agent:run_agentна набореmy_datasetметодом оценки LLM-судьёй. - Сравни
my_agent:v1иmy_agent:v2наdataset.jsonи сохрани отчёт вcomparison_report.json.
Как установить langgraph-testing-evaluation
1Через Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation -a claude-code2Вручную: положить файлы
~/.claude/skills/langgraph-testing-evaluation/SKILL.mdПапка скилла целиком; для одного проекта — .claude/skills/langgraph-testing-evaluation/
1Через Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation -a codex2Вручную: положить файлы
.agents/skills/langgraph-testing-evaluation/SKILL.mdДля всех проектов — ~/.agents/skills/
1Через Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation -a cursor2Вручную: положить файлы
.cursor/skills/langgraph-testing-evaluation/SKILL.mdCursor читает и .agents/skills/, и .claude/skills/
1GitHub CLI
gh skill install soba-labs/langchain-agent-skills skills/langgraph-testing-evaluation2Вручную: положить файлы
.github/skills/langgraph-testing-evaluation/SKILL.md1Через Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation -a gemini-cli2Командой платформы
gemini skills install https://github.com/soba-labs/langchain-agent-skills --path skills/langgraph-testing-evaluationВручную: положить файлы
.devin/skills/langgraph-testing-evaluation/SKILL.mdСтарый путь .windsurf/skills/ тоже читается
Через Skills CLI (Vercel)
npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation -a opencodeClaude.ai
Упакуйте папку в zip и загрузите в Settings → Capabilities → Skills
По шагам, с командами для каждой платформы: Как установить скилл в Claude Code: пять способов
Вопросы о langgraph-testing-evaluation
Как установить langgraph-testing-evaluation в Claude Code?
Запусти npx skills add soba-labs/langchain-agent-skills --skill langgraph-testing-evaluation -a claude-code. Файл скилла находится по пути ~/.claude/skills/langgraph-testing-evaluation/SKILL.md.
Что делает langgraph-testing-evaluation?
Скилл помогает тестировать узлы и взаимодействия в агентах LangGraph и LangChain, оценивать траектории и данные в LangSmith, а также сравнивать версии агента.
Какие языки поддерживает langgraph-testing-evaluation?
Скрипты и рабочие процессы рассчитаны на Python и JavaScript/TypeScript.
Исходный текст
SKILL.md · MIT
---
name: langgraph-testing-evaluation
description: "Use this skill when you need to test or evaluate LangGraph/LangChain agents: writing unit or integration tests, generating test scaffolds, mocking LLM/tool behavior, running trajectory evaluation (match or LLM-as-judge), running LangSmith dataset evaluations, and comparing two agent versions with A/B-style offline analysis. Use it for Python and JavaScript/TypeScript workflows, evaluator design, experiment setup, regression gates, and debugging flaky/incorrect evaluation results."
---
# LangGraph Testing & Evaluation
Practical workflows for validating agent quality with:
- Unit/integration tests
- Trajectory evaluation
- LangSmith dataset evaluations
- A/B-style comparisons between versions
Use this file for high-level flow. Load `references/*` for detailed implementation.
## Start Here
Choose the smallest approach that answers your question:
| Goal | Primary method | Load first |
| --- | --- | --- |
| Validate node logic quickly | Unit tests with mocks | `references/unit-testing-patterns.md` |
| Validate multi-step agent behavior | Trajectory evaluation | `references/trajectory-evaluation.md` |
| Track quality over datasets over time | LangSmith evaluation | `references/langsmith-evaluation.md` |
| Compare old vs new agent versions | A/B comparison | `references/ab-testing.md` |
Recommended order:
1. Unit tests
2. Integration/trajectory checks
3. Dataset evaluation in LangSmith
4. A/B comparison before deployment
## Quick Commands
Run from repo root.
### Generate test scaffolding
```bash
# Python (preferred)
uv run skills/langgraph-testing-evaluation/scripts/generate_test_cases.py my_agent:graph --output tests/ --framework pytest
# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/generate_test_cases.js ./my-agent.ts:graph --output tests/ --framework vitest
```
### Run trajectory evaluation
```bash
# Python: LLM-as-judge
uv run skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.py my_agent:run_agent my_dataset --method llm-judge --model openai:o3-mini
# Python: trajectory match
uv run skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.py my_agent:run_agent dataset.json --method match --trajectory-match-mode strict --reference-trajectory reference.json
# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/run_trajectory_eval.js ./agent.ts:runAgent my_dataset --method llm-judge --model openai:o3-mini --max-concurrency 4
```
### Run LangSmith dataset evaluation
```bash
# Python
uv run skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.py my_agent:run_agent my_dataset --evaluators accuracy,latency --max-concurrency 4
# Python (do not upload experiment results)
uv run skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.py my_agent:run_agent my_dataset --evaluators accuracy --no-upload
# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/evaluate_with_langsmith.js ./agent.ts:runAgent my_dataset --evaluators accuracy,latency --max-concurrency 4
```
### Compare two agent versions
```bash
# Python
uv run skills/langgraph-testing-evaluation/scripts/compare_agents.py my_agent:v1 my_agent:v2 dataset.json --output comparison_report.json
# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/compare_agents.js ./v1.ts:run ./v2.ts:run dataset.json --output comparison_report.json
# JavaScript/TypeScript (force local dataset file only)
node skills/langgraph-testing-evaluation/scripts/compare_agents.js ./v1.ts:run ./v2.ts:run dataset.json --no-langsmith
```
### Create mock response configs
```bash
# Python
uv run skills/langgraph-testing-evaluation/scripts/mock_llm_responses.py create --type sequence --output mock_config.json
# JavaScript/TypeScript
node skills/langgraph-testing-evaluation/scripts/mock_llm_responses.js create --type sequence --output mock_config.json
```
## Core Workflow
1. Define test scope.
- Unit: deterministic logic in one node/function.
- Integration: node interactions and routing.
- End-to-end: complete response quality on realistic inputs.
2. Start from deterministic checks.
- Mock LLM/tool IO for speed and repeatability.
- Keep real-model tests as a smaller, explicit suite.
3. Build/curate dataset examples.
- Use stable inputs and expected outputs.
- Keep schema simple: `inputs` and `outputs` objects (optional `metadata`).
- Compatibility note: scripts also accept singular keys (`input`, `output`) for legacy datasets.
4. Run evaluation with explicit gates.
- Use evaluator keys that map to deployment decisions.
- Set thresholds in CI for regression prevention.
5. Compare versions before rollout.
- Run same dataset on both versions.
- Check both quality and latency.
6. Diagnose failures from traces/experiments.
- Inspect low-scoring examples.
- Split failures by pattern (routing, tool usage, hallucination, latency spikes).
## Current References (Load On Demand)
### `references/unit-testing-patterns.md`
Load when:
- You need node-level and routing test patterns.
- You need pytest/vitest/Jest integration patterns.
- You need robust mocking and flaky-test reduction.
### `references/trajectory-evaluation.md`
Load when:
- You need trajectory match evaluation (`strict`, `unordered`, `subset`, `superset`).
- You need LLM-as-judge trajectory scoring.
- You need LangSmith experiment comparison for trajectory results.
### `references/langsmith-evaluation.md`
Load when:
- You need dataset creation/management in LangSmith.
- You need evaluator signatures and experiment runs in Python/TS.
- You need CI-friendly workflows with quality thresholds.
### `references/ab-testing.md`
Load when:
- You need offline A/B comparison methodology.
- You need significance testing and interpretation.
- You need production traffic split strategy and guardrails.
## Assets
### `assets/templates/test_template.py`
- Runnable Python pytest template aligned with current LangGraph testing patterns.
- Includes:
- Compiled-graph invocation with `thread_id`
- Single-node testing via `compiled_graph.nodes[...]`
- Integration-test placeholder
### `assets/datasets/sample_dataset.json`
- Deterministic seed dataset for LangSmith ingestion.
- Uses `examples: [{ inputs, outputs, metadata }]` format.
### `assets/examples/README.md`
- Documentation-only index for current asset usage.
- Notes where runnable assets live today.
## Script Interface Summary
### `scripts/generate_test_cases.py` / `.js`
Use for fast test scaffolding.
Inputs:
- Graph module path
- Python: `my_module:graph` or `my_module.graph`
- JS/TS: `./file.ts:graph`
Outputs:
- Framework-specific starter tests in target directory.
### `scripts/run_trajectory_eval.py` / `.js`
Use for trajectory scoring with either:
- `--method match`
- `--method llm-judge`
Supports:
- Local dataset files (`.json`)
- LangSmith dataset names
- Optional reference trajectory file with `--reference-trajectory`
- Match modes: `strict`, `unordered`, `subset`, `superset`
Local-only mode:
- `--no-langsmith` in both Python and JavaScript scripts (requires local JSON dataset file)
### `scripts/evaluate_with_langsmith.py` / `.js`
Use for dataset-based evaluation runs and experiment tracking.
Supports:
- Existing dataset by name
- Dataset creation from JSON examples file
- Multiple evaluators (`--evaluators accuracy,latency,...`)
- Concurrency control (`--max-concurrency`)
Python-only:
- `--no-upload` to run without uploading experiment results
### `scripts/compare_agents.py` / `.js`
Use for offline version comparisons:
- Shared dataset input
- Success/latency summaries
- JSON report output for CI artifacts
- Local JSON datasets or LangSmith datasets (JS supports `--no-langsmith` to disable remote loading)
### `scripts/mock_llm_responses.py` / `.js`
Use for deterministic test doubles:
- single
- sequence
- conditional
## Decision Rules
If behavior is deterministic and local:
- Use unit tests first.
If behavior depends on tool sequence/routing:
- Add trajectory evaluation.
If behavior depends on realistic distribution quality:
- Run LangSmith dataset evaluation.
If approving a replacement model/prompt/graph:
- Run A/B comparison and check both quality and latency.
## Common Failure Patterns
### Flaky tests
- Cause: real-model nondeterminism in unit scope.
- Fix: mock LLM/tool calls for unit tests; reserve real-model tests for separate integration marks.
### High trajectory variance
- Cause: overly strict matching for workflows with equivalent paths.
- Fix: switch match mode (`unordered`, `subset`, or `superset`) where appropriate.
### Regressions hidden by averages
- Cause: only aggregate score monitored.
- Fix: inspect per-example failures and segment by category metadata.
### Latency regressions with same quality
- Cause: no explicit latency gate.
- Fix: include latency evaluator and CI threshold.
## Minimal Best Practices
1. Keep fast deterministic tests as the largest share.
2. Version datasets and keep them stable.
3. Track both correctness and latency.
4. Add explicit go/no-go thresholds in CI.
5. Compare candidate vs baseline before production rollout.
6. Investigate failures with trace-level evidence, not only aggregate scores.
© soba-labs · MIT · soba-labs/langchain-agent-skills