Skip to main content
skill CX Codex

ai-data-engineering-rag-pipeline — Claude Code skill: RAG setup

ai-data-engineering-rag-pipeline is a Claude Code skill for building a local RAG workflow with BM25 retrieval, hierarchical document chunks, and retrieval scoring. It tests retrieval against a prepared set of questions.

no license 165 installs ★ 5 updated 2026-10-09

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a claude-code
Open on GitHub

What ai-data-engineering-rag-pipeline does

The skill walks through working with a document corpus and reference questions stored in questions.jsonl. Its search baseline uses a BM25 index built from an inverted index, then measures Recall@10 against relevant documents. This gives you a concrete way to assess retrieval results.

For the RAG pipeline, documents are split into document, section, and paragraph levels, with child chunks linked to parent metadata. The project layout includes Day_01, Day_02, and Day_03; the Day_02 and Day_03 modules install dependencies from their own requirements.txt files. Listed packages include rank-bm25, numpy, sentence-transformers, and faiss-cpu.

You need pip to install the listed dependencies. The relevant modules will not run until those dependencies are installed.

What it helps with

  • prepare reference questions for retrieval tests
  • build a BM25 index over a document corpus
  • measure search results with Recall@10
  • split documents into chunks at several levels
  • attach child chunks to parent metadata

Example prompts for the agent

  • Build a local RAG pipeline with BM25 and evaluate retrieval using questions.jsonl.
  • Split the documents into document, section, and paragraph chunks with parent links.
  • Create reference questions and calculate Recall@10 for this corpus.

How to install ai-data-engineering-rag-pipeline

1Skills CLI (Vercel)

npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a claude-code

2Manually: put the files in place

~/.claude/skills/ai-data-engineering-rag-pipeline/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/ai-data-engineering-rag-pipeline/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about ai-data-engineering-rag-pipeline

How do I install ai-data-engineering-rag-pipeline in Claude Code?

Run npx skills add reason-machines/data-skills --skill ai-data-engineering-rag-pipeline -a claude-code. The skill file is at ~/.claude/skills/ai-data-engineering-rag-pipeline/SKILL.md.

What does ai-data-engineering-rag-pipeline do?

It guides you through a local RAG setup with BM25 retrieval, hierarchical document chunks, and Recall@10 evaluation against reference questions.

Which dependencies are used for the BM25 and RAG modules?

Install dependencies for Day_02 and Day_03 from their separate requirements.txt files. The listed packages include rank-bm25, numpy, sentence-transformers, and faiss-cpu.

Similar skills

All skills for Data and databases →

From the same set82

The whole set →
enterprise-user-management-ai-system enterprise-user-management-ai-system — a skill that scaffolds a user management system with Kanban boards, support tickets and AI analytics for risk, anomalies and burnout. CX Codex +1 372installs enterprise-user-management-system-ai-analytics The enterprise-user-management-system-ai-analytics skill outlines a user management app with task tracking, support tickets, and ML analytics. Use it when planning authentication, admin views, or workload and risk analysis. CX Codex +1 338installs web-analytics-agent-skill web-analytics-agent-skill pulls search and visitor metrics from Google Search Console, GA4, and Bing Webmaster Tools. Use it for SEO checks, comparisons, and reports. CX Codex +1 306installs microsoft-fabric-unified-analytics microsoft-fabric-unified-analytics walks you through a Fabric analytics stack: OneLake layers, a Dataflow Gen2 cleaning step, PySpark gold tables and Power BI on top. CX Codex +1 282installs crypto-etl-analytics-core crypto-etl-analytics-core is a Claude Code skill for building a Python and MySQL cryptocurrency OHLCV pipeline with standard-library analytics. CX Codex +1 273installs enterprise-user-management-system-ai enterprise-user-management-system-ai scaffolds a user management platform pairing React and Node.js with a FastAPI machine learning service. CX Codex +1 249installs zomato-ai-data-engineering-pipeline zomato-ai-data-engineering-pipeline is a Claude Code skill for building a food delivery batch pipeline across S3, Snowflake, dbt, and Airflow, with OpenAI for review enrichment and text-to-SQL. CX Codex +1 128installs high-stakes-analytics-decision-lab The high-stakes-analytics-decision-lab Claude Code skill checks data quality and chooses an analysis method to match the question. It runs for evidence-based reports and decision work that needs uncertainty bounds. CX Codex +1 103installs

Project sponsors

Companies that keep this analytics open