ai-data-engineering-rag-pipeline — Claude Code skill: RAG setup
ai-data-engineering-rag-pipeline is a Claude Code skill for building a local RAG workflow with BM25 retrieval, hierarchical document chunks, and retrieval scoring. It tests retrieval against a prepared set of questions.
no license 165 installs ★ 5 updated 2026-10-09
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a claude-codeWhat ai-data-engineering-rag-pipeline does
The skill walks through working with a document corpus and reference questions stored in questions.jsonl. Its search baseline uses a BM25 index built from an inverted index, then measures Recall@10 against relevant documents. This gives you a concrete way to assess retrieval results.
For the RAG pipeline, documents are split into document, section, and paragraph levels, with child chunks linked to parent metadata. The project layout includes Day_01, Day_02, and Day_03; the Day_02 and Day_03 modules install dependencies from their own requirements.txt files. Listed packages include rank-bm25, numpy, sentence-transformers, and faiss-cpu.
You need pip to install the listed dependencies. The relevant modules will not run until those dependencies are installed.
What it helps with
- prepare reference questions for retrieval tests
- build a BM25 index over a document corpus
- measure search results with Recall@10
- split documents into chunks at several levels
- attach child chunks to parent metadata
Example prompts for the agent
- Build a local RAG pipeline with BM25 and evaluate retrieval using
questions.jsonl. - Split the documents into document, section, and paragraph chunks with parent links.
- Create reference questions and calculate Recall@10 for this corpus.
How to install ai-data-engineering-rag-pipeline
1Skills CLI (Vercel)
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a claude-code2Manually: put the files in place
~/.claude/skills/ai-data-engineering-rag-pipeline/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/ai-data-engineering-rag-pipeline/
1Skills CLI (Vercel)
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a codex2Manually: put the files in place
.agents/skills/ai-data-engineering-rag-pipeline/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a cursor2Manually: put the files in place
.cursor/skills/ai-data-engineering-rag-pipeline/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install reason-machines/data-skills skills/ai-data-engineering-rag-pipeline2Manually: put the files in place
.github/skills/ai-data-engineering-rag-pipeline/SKILL.md1Skills CLI (Vercel)
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a gemini-cli2Platform CLI
gemini skills install https://github.com/reason-machines/data-skills --path skills/ai-data-engineering-rag-pipelineManually: put the files in place
.devin/skills/ai-data-engineering-rag-pipeline/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about ai-data-engineering-rag-pipeline
How do I install ai-data-engineering-rag-pipeline in Claude Code?
Run npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a claude-code. The skill file is at ~/.claude/skills/ai-data-engineering-rag-pipeline/SKILL.md.
What does ai-data-engineering-rag-pipeline do?
It guides you through a local RAG setup with BM25 retrieval, hierarchical document chunks, and Recall@10 evaluation against reference questions.
Which dependencies are used for the BM25 and RAG modules?
Install dependencies for Day_02 and Day_03 from their separate requirements.txt files. The listed packages include rank-bm25, numpy, sentence-transformers, and faiss-cpu.