skill CX Codex

rag-eval — Claude Code skill: what it does, how to install

Q: How do I install rag-eval in Claude Code?

Run `npx skills add NVIDIA/skills --skill rag-eval -a claude-code`, then run `npx skills add NVIDIA/skills --skill rag-eval -a claude-code` again. The skill file is `~/.claude/skills/rag-eval/SKILL.md`.

Q: What does rag-eval do?

It evaluates filesystem RAG datasets through `scripts/eval/evaluate_rag.py` and RAGAS metrics. The skill covers dataset preparation, quality tuning, result analysis, and error triage.

Q: What does rag-eval require to run?

You need a repository checkout, Python `3.11+`, `uv`, and reachable RAG plus ingestor endpoints. RAGAS also requires `NVIDIA_API_KEY`.

Evaluates filesystem RAG datasets built around corpus/ and train.json. Use it to compare retrieval and generation settings with RAGAS quality scores.

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓

npx skills add NVIDIA/skills --skill rag-eval -a claude-code

Open on GitHub

What rag-eval does

This skill guides filesystem RAG evaluation with corpus/ and train.json, then runs scripts/eval/evaluate_rag.py and reads RAGAS JSON output. It covers retrieval and generation comparisons through flags such as --top_k, --vdb_top_k, and --temperature.

Run the commands from the repository root. The evaluator needs Python 3.11+, uv, reachable RAG and ingestor endpoints, plus NVIDIA_API_KEY for RAGAS; install its dependencies with uv sync --project scripts/eval.

It also helps trace network failures, empty contexts, and collection mismatches. This is a poor fit for latency or throughput testing because the skill directs those benchmarks to rag-perf.

What it helps with

prepare a dataset with corpus/ and train.json
run a RAGAS evaluation with evaluate_rag.py
compare retrieval and generation settings
trace empty generated_contexts
diagnose ingestor and NVIDIA_API_KEY errors

Example prompts for the agent

Run a RAGAS evaluation for /path/to/my_dataset with uv run --project scripts/eval python scripts/eval/evaluate_rag.py.
Investigate empty generated_contexts and check the collection plus --vdb_top_k.
Compare the RAGAS results after changing --top_k and --temperature.

How to install rag-eval

1Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill rag-eval -a claude-code

2Manually: put the files in place

~/.claude/skills/rag-eval/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/rag-eval/

1Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill rag-eval -a codex

2Manually: put the files in place

.agents/skills/rag-eval/SKILL.md

For all projects use ~/.agents/skills/

1Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill rag-eval -a cursor

2Manually: put the files in place

.cursor/skills/rag-eval/SKILL.md

Cursor also reads .agents/skills/ and .claude/skills/

1GitHub CLI

gh skill install NVIDIA/skills rag-eval

2Manually: put the files in place

.github/skills/rag-eval/SKILL.md

1Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill rag-eval -a gemini-cli

2Platform CLI

gemini skills install https://github.com/NVIDIA/skills --path skills/rag-eval

Manually: put the files in place

.devin/skills/rag-eval/SKILL.md

The legacy .windsurf/skills/ path still works

Skills CLI (Vercel)

npx skills add NVIDIA/skills --skill rag-eval -a opencode

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about rag-eval

How do I install rag-eval in Claude Code?

Run npx skills add NVIDIA/skills --skill rag-eval -a claude-code, then run npx skills add NVIDIA/skills --skill rag-eval -a claude-code again. The skill file is ~/.claude/skills/rag-eval/SKILL.md.

What does rag-eval do?

It evaluates filesystem RAG datasets through scripts/eval/evaluate_rag.py and RAGAS metrics. The skill covers dataset preparation, quality tuning, result analysis, and error triage.

What does rag-eval require to run?

You need a repository checkout, Python 3.11+, uv, and reachable RAG plus ingestor endpoints. RAGAS also requires NVIDIA_API_KEY.

Source text

SKILL.md · Apache-2.0

---
name: rag-eval
version: "2.6.0"
description: >-
  Filesystem RAG benchmarks: corpus/, train.json, evaluate_rag.py (RAGAS quality). Not for prod
  monitoring, latency/throughput benchmarking (use rag-perf), or evals outside this repo layout.
license: Apache-2.0
compatibility: Repository checkout with uv; Python 3.11+; run from repo root; uv sync --project scripts/eval (eval deps live in scripts/eval/pyproject.toml); network to RAG, ingestor, and vdb endpoints; NVIDIA_API_KEY for RAGAS; optional RAG_EVAL_JUDGE_MODEL (default mistralai/mixtral-8x22b-instruct-v0.1).
metadata:
  author: NVIDIA RAG <[email protected]>
  github-url: "https://github.com/NVIDIA-AI-Blueprints/rag"
  endpoint-openapi-schemas:
    - docs/api_reference/openapi_schema_rag_server.json
    - docs/api_reference/openapi_schema_ingestor_server.json
  argument-hint: RAGAS eval | evaluate_rag | train.json | corpus | results json | error triage | uv run --project scripts/eval | enable_reranker | query_rewriting | temperature | skip_ingestion
  tags:
    - nvidia
    - blueprint
    - rag
    - evaluation
    - ragas
    - benchmarking
    - nvidia-rag-blueprint
  languages:
    - python
    - shell
  frameworks:
    - ragas
    - fastapi
  domain: ai-ml
allowed-tools: Read Grep Glob Bash(ls *) Bash(python3 *) Bash(uv *) Write Edit
---

# On-disk RAG evaluation (`corpus/` + `train.json`)

## Purpose

Guide agents through NVIDIA RAG Blueprint **filesystem** benchmarks: preparing `corpus/` and `train.json`, running `scripts/eval/evaluate_rag.py`, tuning retrieval and generation flags for **quality** comparisons, interpreting RAGAS JSON outputs, and triaging failures (HTTP/stream errors, empty contexts, collection mismatch, judge API).

For **latency, throughput, and load testing**, use the **rag-perf** skill (`scripts/rag-perf`, `docs/performance-benchmarking.md`) — not this skill.

## When not to use

Do **not** use this skill for: deploying or repairing services (use rag-blueprint); evaluating APIs without the `corpus/` + `train.json` layout; general ML experimentation unrelated to this evaluator; production monitoring/alerting; or latency/throughput benchmarking (use **rag-perf**).

## Prerequisites

- Repo cloned; **run commands from repo root** (imports and paths assume this).
- Python **3.11+** and **uv**; eval deps: `uv sync --project scripts/eval`.
- Reachable **RAG server** and **ingestor** (defaults often `localhost:8081` / `8082`).
- **`NVIDIA_API_KEY`** for RAGAS (see [credential hygiene](references/benchmark-execution.md#credential-hygiene-nvidia_api_key)); optional **`RAG_EVAL_JUDGE_MODEL`**.
- Dataset roots passed to `--dataset-paths` each contain **`corpus/`** and **`train.json`**.

## Instructions

1. **Prepare data** — Ensure each dataset directory matches the layout and `train.json` rules in [`references/dataset-and-conversion.md`](references/dataset-and-conversion.md). When sources arrive as public links (sites or dataset pages), materialize documents under `corpus/`—prefer **PDF** for multimodal content so **images stay embedded**; convert CSV/JSONL/etc. using the patterns there.
2. **Run eval** — `uv run --project scripts/eval python scripts/eval/evaluate_rag.py` with `--dataset-paths`, `--host`, and `--port`. See [`references/benchmark-execution.md`](references/benchmark-execution.md) for command examples, outputs, and errors. Use [`references/evaluate-rag-cli.md`](references/evaluate-rag-cli.md) for flag-level detail.
3. **Tune quality** — Adjust `--top_k` / `--vdb_top_k`, reranker and query-rewriting toggles, and generation overrides (`--temperature`, `--top-p`, `--max-tokens`) as documented in [`references/benchmark-execution.md`](references/benchmark-execution.md) when comparing retrieval/generation configs for RAGAS scores.
4. **Analyze results** — Use [`references/result-analysis.md`](references/result-analysis.md) for scripts; scan `rag_*_evaluation_summary.json` for headline RAGAS metrics.
5. **Triage errors** — Use the [error signal table](references/benchmark-execution.md#common-error-cases-and-signals) and the **Troubleshooting** section below.

## Examples

**Set API key without putting secrets in shell history (preferred patterns):** load from a gitignored env file or secrets manager; avoid committing `.env`; rotate keys if exposed. Details: [`references/benchmark-execution.md#credential-hygiene-nvidia_api_key`](references/benchmark-execution.md#credential-hygiene-nvidia_api_key).

**Minimal eval (key already in environment):**

```bash
uv sync --project scripts/eval
uv run --project scripts/eval python scripts/eval/evaluate_rag.py \
  --dataset-paths /path/to/my_dataset \
  --host localhost \
  --port 8081
```

**Pretty-print summary JSON:**

```bash
python3 -m json.tool results/my_dataset/rag_my_dataset_evaluation_summary.json
```

More examples (skip ingestion, quality sweeps): [`references/benchmark-execution.md`](references/benchmark-execution.md).

## Limitations

- Evaluator behavior is fixed to the **filesystem contract** and `evaluate_rag.py`; it does not substitute for custom offline judges or non-RAG benchmarks.
- **Vector DB / embedding** choices follow deployed ingestor and RAG env — not overridden by this CLI alone.
- **Scores depend on** retrieval quality, judge model availability, and `NVIDIA_API_KEY`; empty contexts yield partial RAGAS metrics (see references).
- Large procedural detail lives under **`references/`** to keep routing concise; read those files when the user needs step-by-step conversion, full flags, or error tables.

## Troubleshooting

| Error / signal | Likely cause | What to do |
|----------------|--------------|------------|
| Immediate exit mentioning `NVIDIA_API_KEY` | Missing or invalid key | Set key via secure channel; see credential hygiene in [`references/benchmark-execution.md`](references/benchmark-execution.md). |
| `train.json must be a JSON array` | Wrong JSON shape | Top-level array of objects; validate per [`references/dataset-and-conversion.md`](references/dataset-and-conversion.md). |
| Fewer rows in `evaluation_data.json` than `train.json` | Per-query failures | Check stderr: network or stream JSON errors; see error table in benchmark-execution. |
| Empty `generated_contexts` everywhere | Retrieval gap | Verify collection, ingestion, `top_k` / `vdb_top_k`, and `ingestor_server_url` **without** `/v1` suffix. |
| Ingestor 404 on upload | Bad ingestor base URL | Pass `http://host:port` only — code appends `/v1/`. |

Full signal table: [`references/benchmark-execution.md#common-error-cases-and-signals`](references/benchmark-execution.md#common-error-cases-and-signals).

## Gotchas

- **Run from repo root**: paths and imports in `scripts/eval/evaluate_rag.py` assume this; a wrong directory silently breaks imports.
- **`--ingestor_server_url`**: pass `http://host:port` without `/v1`—the code appends `/v1/` automatically. Including `/v1` causes 404s on ingestor calls.
- **Vector DB / embedding settings**: not set by this CLI; configure via the deployed ingestor and RAG server env vars (e.g. `APP_VECTORSTORE_URL`, embedding model).
- **`--model` / `--llm_endpoint`**: forwarded verbatim only when explicitly set; omit to keep the server's configured LLM.
- **Stale collections**: a previous run's ingested data persists unless you use `--force_ingestion`. Use `--collection` with a unique name when comparing quality across isolated runs.
- **Empty context metrics**: if all `generated_contexts` are empty, RAGAS scores only `nv_accuracy` and leaves the other two metrics blank—this is not a silent success.

## Source of truth

| Piece | Location |
|-------|----------|
| Driver | `scripts/eval/evaluate_rag.py` (`CORPUS_DIRECTORY` = `corpus`, `EVAL_DATA` = `train.json`) |
| Human README (always in-repo) | `scripts/eval/README.md` |
| Full CLI (flags, defaults) | `scripts/eval/evaluate_rag.py --help`; [`references/evaluate-rag-cli.md`](references/evaluate-rag-cli.md) |
| Dataset / conversion | [`references/dataset-and-conversion.md`](references/dataset-and-conversion.md) |
| Runs, outputs, errors | [`references/benchmark-execution.md`](references/benchmark-execution.md) |
| Result analysis scripts | [`references/result-analysis.md`](references/result-analysis.md) |
| Latency / throughput | **rag-perf** skill, `docs/performance-benchmarking.md` |

## Agent playbook

1. **Run eval** — `uv sync --project scripts/eval` then `uv run --project scripts/eval python scripts/eval/evaluate_rag.py` with required `--dataset-paths`, `--host`, and `--port` (and env `NVIDIA_API_KEY`). Argument `--ingestor_server_url` is optional (defaults to `http://localhost:8082`); pass it only when overriding the ingestor endpoint.
2. **Quality tuning** — See [`references/benchmark-execution.md`](references/benchmark-execution.md): `--top_k`/`--vdb_top_k`, reranker and query-rewriting toggles, `--temperature`, `--top-p`, `--max-tokens`.
3. **Data conversion** — Follow [`references/dataset-and-conversion.md`](references/dataset-and-conversion.md).
4. **Analyze results** — [`references/result-analysis.md`](references/result-analysis.md); quick scan: `python3 -m json.tool results/<dataset>/rag_<dataset>_evaluation_summary.json`.
5. **Error triage** — [`references/benchmark-execution.md#common-error-cases-and-signals`](references/benchmark-execution.md#common-error-cases-and-signals).

Similar skills

Everything for AI and ML →

rag-observability-evals devops-skills A skill for monitoring and evaluating RAG systems: retrieval quality metrics, groundedness checks, hallucination detection and regression testing. CX Codex +1 318installs langchain-rag langchain-ai Equips coding agents with reference RAG implementations in Python and TypeScript using LangChain loaders, splitters, and vector stores. CX Codex +1 17,500installs agent-platform-rag-engine-management official google Use this skill to inspect Agent Platform RAG Engine corpora, find their files, retrieve context, and draft grounded answers. It fits workflows built around RAG Engine in Google Cloud. CX Codex 8,626installs rag-infrastructure devops-skills A skill for building RAG infrastructure: document ingestion, embeddings, vector stores, hybrid search and grounded LLM answers. CX Codex +1 325installs pinecone-rag official github This skill guides Pinecone-based RAG pipelines and persistent agent memory. Use it for semantic document search, hybrid retrieval, or context shared across sessions. 24installs evaluate-rag ai-evals-course Guides separate evaluation of RAG retrieval and generation, metric selection, and document chunking tests. CX Codex +1 970installs

From the same set178

cuopt-install Installs NVIDIA cuOpt for Python, C, or as a REST server via pip, conda, or Docker, then verifies the setup. Helpful when configuring GPU-accelerated routing environments. CX Codex +1 2,500installs data-designer Creates synthetic datasets and Python data-generation pipelines with Data Designer. Use it when you need to describe a dataset and turn that description into a script. CX Codex +1 2,500installs deepstream-dev Teaches the agent to write NVIDIA DeepStream SDK code in Python: GStreamer pipelines, TensorRT inference, tracking and broker output. CX Codex +1 2,400installs nemo-retriever Indexes and queries document collections using the NeMo Retriever 26.8.1 CLI. Useful for local LanceDB storage and deployed Retriever services. CX Codex +1 2,400installs cuopt-routing-api-python A skill that walks your agent through vehicle routing with the cuOpt Python API: TSP, VRP and pickup-delivery. Useful when the solver runs in Python. CX Codex +1 2,300installs dynamo-recipe-runner Discovers, validates, patches, and deploys NVIDIA Dynamo Kubernetes recipes for model serving across vLLM or SGLang runtimes. CX Codex +1 2,300installs deepstream-import-vision-model This skill takes an object-detection model from HuggingFace or NGC through a full NVIDIA DeepStream pipeline: ONNX, TRT engine, bbox parser, benchmark, PDF report. CX Codex +1 2,300installs omniverse-usd-performance-tuning A skill for diagnosing and optimizing USD scenes: slow loading, low FPS, high memory. It owns the user-facing workflow across phases. CX Codex +1 2,300installs