# ai-data-engineering-rag-pipeline

ai-data-engineering-rag-pipeline is a Claude Code skill for building a local RAG workflow with BM25 retrieval, hierarchical document chunks, and retrieval scoring. It tests retrieval against a prepared set of questions.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 165 installs · ★ 5 repo stars
- Author: reason-machines
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/data-skills/tree/HEAD/skills/ai-data-engineering-rag-pipeline/SKILL.md
- Updated: 2026-10-09
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/ai-data-engineering-rag-pipeline

## What it does

The skill walks through working with a document corpus and reference questions stored in `questions.jsonl`. Its search baseline uses a BM25 index built from an inverted index, then measures Recall@10 against relevant documents. This gives you a concrete way to assess retrieval results.

For the RAG pipeline, documents are split into document, section, and paragraph levels, with child chunks linked to parent metadata. The project layout includes `Day_01`, `Day_02`, and `Day_03`; the `Day_02` and `Day_03` modules install dependencies from their own `requirements.txt` files. Listed packages include `rank-bm25`, `numpy`, `sentence-transformers`, and `faiss-cpu`.

You need `pip` to install the listed dependencies. The relevant modules will not run until those dependencies are installed.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/ai-data-engineering-rag-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/ai-data-engineering-rag-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/ai-data-engineering-rag-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/ai-data-engineering-rag-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/data-skills skills/ai-data-engineering-rag-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/ai-data-engineering-rag-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/data-skills --path skills/ai-data-engineering-rag-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/ai-data-engineering-rag-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/data-skills/tree/HEAD/skills/ai-data-engineering-rag-pipeline/SKILL.md

## When to use it
- prepare reference questions for retrieval tests
- build a BM25 index over a document corpus
- measure search results with Recall@10
- split documents into chunks at several levels
- attach child chunks to parent metadata

## Example prompts
- Build a local RAG pipeline with BM25 and evaluate retrieval using `questions.jsonl`.
- Split the documents into document, section, and paragraph chunks with parent links.
- Create reference questions and calculate Recall@10 for this corpus.

## Similar
- [rag-architect](https://affpapa.org/en/nevibecode/skills/borghei-rag-architect.md) — skill · 137 installs
- [rag-implementer](https://affpapa.org/en/nevibecode/skills/oakoss-rag-implementer.md) — skill · 121 installs
- [embedding-pipeline-builder](https://affpapa.org/en/nevibecode/skills/embedding-pipeline-builder.md) — skill · 259 installs
- [evaluate-rag](https://affpapa.org/en/nevibecode/skills/evaluate-rag.md) — skill · 1100 installs
- [dspy-rag-pipeline](https://affpapa.org/en/nevibecode/skills/dspy-rag-pipeline.md) — skill · ★ 125 repo stars
- [rag](https://affpapa.org/en/nevibecode/skills/rag.md) — skill · 3900 installs

From the same set: [reason-machines/data-skills](https://affpapa.org/en/nevibecode/by/reason-machines/data-skills.md) — 82 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
