Skip to main content
skill CX Codex

data-pipeline-builder — Claude Code skill: what it does

data-pipeline-builder turns source, destination and transformation notes into a runnable ETL/ELT pipeline: Python or SQL code, an Airflow DAG, quality checks and a spec file.

MIT 252 installs ★ 440 updated 2026-10-20

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a claude-code
Open on GitHub

What data-pipeline-builder does

The skill walks from requirements to files. It asks which sources feed the pipeline, where the data lands, what has to be transformed and how fresh it must be. Then it designs the layout and generates the code. Five reference documents back this up: project structure, Python patterns, quality checks, orchestration config, spec template.

You see the design first. Architecture, source list, schedule and quality gates need your approval before any code appears, which keeps you from rewriting a pile of generated classes. What comes out: extractors built on BaseExtractor, transformers in Python or SQL, loaders on BaseLoader, a wired Airflow DAG, a YAML pipeline config, tests and data-pipeline-spec.md.

The operating rules cover re-runnable steps, watermarks for incremental loads, a dead letter path for records that fail processing and handling of source schema drift. Credentials go through environment variables or a secret manager, never into the code. Bash sits in the tool list, so the skill runs tests and environment commands itself. Logic that fits in SQL stays in SQL.

What it helps with

  • Build a pipeline across several sources without breaking on reruns
  • Sketch the warehouse schema with partitioning and clustering before code exists
  • Add data quality checks with alert thresholds
  • Set up incremental loads with watermarks and dead letter handling
  • Get a pipeline spec with a backfill plan and failure runbook

Example prompts for the agent

  • Build an ELT pipeline from the orders table in Postgres to ClickHouse, refreshed hourly
  • Show me the pipeline design and transformations first, generate code after I approve
  • Add quality checks and a dead letter path for events from the /v1/events API

How to install data-pipeline-builder

1Skills CLI (Vercel)

npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a claude-code

2Manually: put the files in place

~/.claude/skills/data-pipeline-builder/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/data-pipeline-builder/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about data-pipeline-builder

How to install data-pipeline-builder in Claude Code?

Run npx skills add OneWave-AI/claude-skills --skill data-pipeline-builder -a claude-code. The skill lands at ~/.claude/skills/data-pipeline-builder/SKILL.md.

What does data-pipeline-builder do?

It takes your sources, destination and transformation requirements and returns pipeline code with orchestration, quality checks and a written spec.

Which stack does the skill support?

It asks about technology preferences such as Airflow, dbt or Spark and picks components to match. Orchestration is delivered as an Airflow DAG with a YAML config.

Source text

SKILL.md · MIT
---
name: data-pipeline-builder
description: Designs and builds ETL/ELT data pipelines. Takes data sources, destination, transformation requirements. Generates pipeline code (Python/SQL), scheduling config, error handling, monitoring setup, and data quality checks. Outputs data-pipeline-spec.md + implementation files.
tools: Read, Write, Edit, Glob, Grep, Bash
model: inherit
---

# Data Pipeline Builder

Design and implement production-grade ETL/ELT data pipelines: take data sources, a destination, and transformation requirements, then produce a complete pipeline specification plus all implementation files needed to run it.

## Contents

- `references/project-structure.md` -- output file layout, architecture pattern selection, component selection.
- `references/python-patterns.md` -- Python code standards and base extractor/transformer/loader/retry patterns.
- `references/quality-checks.md` -- composable data quality check framework and built-in checks.
- `references/orchestration-config.md` -- Airflow DAG, pipeline config YAML, and monitoring/alerting patterns.
- `references/spec-template.md` -- the `data-pipeline-spec.md` output template.

## Workflow

1. Gather requirements. If the user gave clear requirements, proceed to design. Otherwise ask targeted questions: data sources (databases, APIs, files, streams); destination (warehouse, lake, database); transformations (joins, aggregations, filters, business rules); freshness requirement (real-time, hourly, daily); technology preferences (Airflow, dbt, Spark, cloud provider); data quality and compliance requirements.

2. Analyze and design. Catalog each source (connection type, auth, schema, volume, CDC availability, rate limits). Define the destination (platform, schema design, partitioning, clustering, access patterns). Map transformations (field mappings, business logic, type conversions, joins, aggregations, deduplication, SCD handling, derived fields). Establish non-functional requirements (freshness SLA, processing window, failure tolerance, retention, compliance). Select an architecture pattern and components per `references/project-structure.md`.

3. Present the design before generating code. Confirm architecture, sources, destination, schedule, key transformations, and quality gates with the user, then proceed on approval.

4. Generate implementation. Produce all files following the layout in `references/project-structure.md`, customized to the specific pipeline with no placeholder code requiring manual editing:
   - For each source, generate a concrete extractor inheriting from `BaseExtractor` (see `references/python-patterns.md`).
   - For each transformation, generate a concrete transformer class or SQL file.
   - For each destination, generate a concrete loader inheriting from `BaseLoader`.
   - Generate the Airflow DAG with all task dependencies wired up and the pipeline config YAML (see `references/orchestration-config.md`).
   - Generate quality checks tailored to the data and monitoring config with appropriate alert thresholds (see `references/quality-checks.md` and `references/orchestration-config.md`).
   - Generate tests for all custom business logic.

5. Generate the specification last. Produce `data-pipeline-spec.md` using `references/spec-template.md`, referencing all implementation files and incorporating design decisions made during the process.

## Operating Rules

- Design for idempotency -- make every step safely re-runnable.
- Include watermark/checkpoint tracking for incremental pipelines.
- Include dead letter handling for records that fail processing.
- Include schema evolution handling -- sources will change their schemas.
- Never hardcode credentials -- use environment variables or secret managers.
- Never skip quality checks -- they are the first line of defense against bad data.
- Prefer SQL for transformations expressible in SQL; use Python for complex logic that does not map cleanly to SQL.
- Include a backfill strategy and an operational runbook covering common failure scenarios in the spec.
- Use structured logging throughout and track data lineage at every transformation step.

© OneWave-AI · MIT · OneWave-AI/claude-skills

Similar skills

All skills for Data and databases →

From the same set185

The whole set →
cowork-deal-room The cowork-deal-room skill reviews deal-room files in five stages and produces risk findings, term comparisons, and negotiation guidance. CX Codex +1 253installs lead-scoring-model This skill analyzes your sales history and builds a lead-scoring model tied to your actual deal data. It generates documentation with scoring dimensions, thresholds, and a CRM implementation guide. CX Codex +1 253installs incident-responder The incident-responder skill investigates production incidents by checking logs and recent changes, classifying severity, and preparing recovery guidance and a report. Use it during an outage and for follow-up. CX Codex +1 253installs runbook-generator The runbook-generator skill inspects a project’s code and deployment configuration, then writes an operational runbook.md. Use it to document releases, incident response, or disaster recovery. CX Codex +1 252installs roi-calculator roi-calculator turns your current costs, manual hours and hourly rates into a written ROI report on an AI project, with payback and sensitivity cases. CX Codex +1 252installs investor-update-writer investor-update-writer generates monthly and quarterly updates for startup investors into a structured investor-update.md file using metrics, milestones, and runway. CX Codex +1 252installs onboarding-checklist Generates phased client onboarding checklists in onboarding-checklist.md with task ownership, criteria, and five email templates for SaaS, agencies, or consulting. CX Codex +1 251installs multi-agent-client-onboarding multi-agent-client-onboarding runs three parallel agents to audit workflows, map a client’s tech stack, and draft an AI roadmap. It is built for structured client assessments. CX Codex +1 251installs

Project sponsors

Companies that keep this analytics open