data-pipeline-builder — Claude Code skill: what it does
data-pipeline-builder turns source, destination and transformation notes into a runnable ETL/ELT pipeline: Python or SQL code, an Airflow DAG, quality checks and a spec file.
MIT 252 installs ★ 440 updated 2026-10-20
npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a claude-codeWhat data-pipeline-builder does
The skill walks from requirements to files. It asks which sources feed the pipeline, where the data lands, what has to be transformed and how fresh it must be. Then it designs the layout and generates the code. Five reference documents back this up: project structure, Python patterns, quality checks, orchestration config, spec template.
You see the design first. Architecture, source list, schedule and quality gates need your approval before any code appears, which keeps you from rewriting a pile of generated classes. What comes out: extractors built on BaseExtractor, transformers in Python or SQL, loaders on BaseLoader, a wired Airflow DAG, a YAML pipeline config, tests and data-pipeline-spec.md.
The operating rules cover re-runnable steps, watermarks for incremental loads, a dead letter path for records that fail processing and handling of source schema drift. Credentials go through environment variables or a secret manager, never into the code. Bash sits in the tool list, so the skill runs tests and environment commands itself. Logic that fits in SQL stays in SQL.
What it helps with
- Build a pipeline across several sources without breaking on reruns
- Sketch the warehouse schema with partitioning and clustering before code exists
- Add data quality checks with alert thresholds
- Set up incremental loads with watermarks and dead letter handling
- Get a pipeline spec with a backfill plan and failure runbook
Example prompts for the agent
- Build an ELT pipeline from the
orderstable in Postgres to ClickHouse, refreshed hourly - Show me the pipeline design and transformations first, generate code after I approve
- Add quality checks and a dead letter path for events from the
/v1/eventsAPI
How to install data-pipeline-builder
1Skills CLI (Vercel)
npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a claude-code2Manually: put the files in place
~/.claude/skills/data-pipeline-builder/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/data-pipeline-builder/
1Skills CLI (Vercel)
npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a codex2Manually: put the files in place
.agents/skills/data-pipeline-builder/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a cursor2Manually: put the files in place
.cursor/skills/data-pipeline-builder/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install OneWave-AI/claude-skills data-pipeline-builder2Manually: put the files in place
.github/skills/data-pipeline-builder/SKILL.md1Skills CLI (Vercel)
npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a gemini-cli2Platform CLI
gemini skills install https://github.com/OneWave-AI/claude-skills --path data-pipeline-builderManually: put the files in place
.devin/skills/data-pipeline-builder/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about data-pipeline-builder
How to install data-pipeline-builder in Claude Code?
Run npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a claude-code. The skill lands at ~/.claude/skills/data-pipeline-builder/SKILL.md.
What does data-pipeline-builder do?
It takes your sources, destination and transformation requirements and returns pipeline code with orchestration, quality checks and a written spec.
Which stack does the skill support?
It asks about technology preferences such as Airflow, dbt or Spark and picks components to match. Orchestration is delivered as an Airflow DAG with a YAML config.
Source text
SKILL.md · MIT
--- name: data-pipeline-builder description: Designs and builds ETL/ELT data pipelines. Takes data sources, destination, transformation requirements. Generates pipeline code (Python/SQL), scheduling config, error handling, monitoring setup, and data quality checks. Outputs data-pipeline-spec.md + implementation files. tools: Read, Write, Edit, Glob, Grep, Bash model: inherit --- # Data Pipeline Builder Design and implement production-grade ETL/ELT data pipelines: take data sources, a destination, and transformation requirements, then produce a complete pipeline specification plus all implementation files needed to run it. ## Contents - `references/project-structure.md` -- output file layout, architecture pattern selection, component selection. - `references/python-patterns.md` -- Python code standards and base extractor/transformer/loader/retry patterns. - `references/quality-checks.md` -- composable data quality check framework and built-in checks. - `references/orchestration-config.md` -- Airflow DAG, pipeline config YAML, and monitoring/alerting patterns. - `references/spec-template.md` -- the `data-pipeline-spec.md` output template. ## Workflow 1. Gather requirements. If the user gave clear requirements, proceed to design. Otherwise ask targeted questions: data sources (databases, APIs, files, streams); destination (warehouse, lake, database); transformations (joins, aggregations, filters, business rules); freshness requirement (real-time, hourly, daily); technology preferences (Airflow, dbt, Spark, cloud provider); data quality and compliance requirements. 2. Analyze and design. Catalog each source (connection type, auth, schema, volume, CDC availability, rate limits). Define the destination (platform, schema design, partitioning, clustering, access patterns). Map transformations (field mappings, business logic, type conversions, joins, aggregations, deduplication, SCD handling, derived fields). Establish non-functional requirements (freshness SLA, processing window, failure tolerance, retention, compliance). Select an architecture pattern and components per `references/project-structure.md`. 3. Present the design before generating code. Confirm architecture, sources, destination, schedule, key transformations, and quality gates with the user, then proceed on approval. 4. Generate implementation. Produce all files following the layout in `references/project-structure.md`, customized to the specific pipeline with no placeholder code requiring manual editing: - For each source, generate a concrete extractor inheriting from `BaseExtractor` (see `references/python-patterns.md`). - For each transformation, generate a concrete transformer class or SQL file. - For each destination, generate a concrete loader inheriting from `BaseLoader`. - Generate the Airflow DAG with all task dependencies wired up and the pipeline config YAML (see `references/orchestration-config.md`). - Generate quality checks tailored to the data and monitoring config with appropriate alert thresholds (see `references/quality-checks.md` and `references/orchestration-config.md`). - Generate tests for all custom business logic. 5. Generate the specification last. Produce `data-pipeline-spec.md` using `references/spec-template.md`, referencing all implementation files and incorporating design decisions made during the process. ## Operating Rules - Design for idempotency -- make every step safely re-runnable. - Include watermark/checkpoint tracking for incremental pipelines. - Include dead letter handling for records that fail processing. - Include schema evolution handling -- sources will change their schemas. - Never hardcode credentials -- use environment variables or secret managers. - Never skip quality checks -- they are the first line of defense against bad data. - Prefer SQL for transformations expressible in SQL; use Python for complex logic that does not map cleanly to SQL. - Include a backfill strategy and an operational runbook covering common failure scenarios in the spec. - Use structured logging throughout and track data lineage at every transformation step.
© OneWave-AI · MIT · OneWave-AI/claude-skills