# etl-pipeline

etl-pipeline is a skill for building construction-data workflows that extract, clean, validate, aggregate, and send results to files, databases, APIs, or dashboards.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 117 installs · ★ 434 repo stars
- Author: datadrivenconstruction
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/tree/HEAD/2_DDC_Book/4.2-ETL-Automation/etl-pipeline/SKILL.md
- Updated: 2026-10-19
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/datadrivenconstruction-etl-pipeline

## What it does

This skill guides agents through ETL workflows for construction data. Its examples extract tables from PDFs, combine Excel files, read from APIs or databases, clean and validate records, and produce summaries for outputs such as Excel, PDF, databases, APIs, and dashboards.

The examples use Python and show separate Extract, Transform, and Load steps. Validation checks can flag missing columns, non-positive values, nulls, and duplicates; aggregation can follow project, building, and level hierarchies. The skill requires `python3` and demonstrates `pandas`, `pdfplumber`, `requests`, and `sqlite3`. PDF table extraction depends on tables being readable by the shown parser.

## Requirements
- Works with: n8n

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill etl-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/etl-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/etl-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill etl-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/etl-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill etl-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/etl-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction 2_DDC_Book/4.2-ETL-Automation/etl-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/etl-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill etl-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --path 2_DDC_Book/4.2-ETL-Automation/etl-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/etl-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction --skill etl-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/tree/HEAD/2_DDC_Book/4.2-ETL-Automation/etl-pipeline/SKILL.md

## When to use it
- extract tables from PDF documents
- combine data from multiple Excel files
- clean and validate construction records
- aggregate quantities and costs by project hierarchy
- export processed results to Excel or a database

## Example prompts
- Create a workflow that reads PDF tables, cleans the records, and saves the result to Excel.
- Combine the Excel files in this folder, validate `Volume` and `ElementId`, then group the data by project.
- Read project data from an API, calculate category totals, and prepare a dashboard dataset.

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: ba18bb7950e64bfe.

BEGIN THIRD-PARTY FILE ba18bb7950e64bfe
````text
---
name: "etl-pipeline"
description: "Build automated ETL (Extract-Transform-Load) pipelines for construction data. Process PDFs, Excel, BIM exports. Generate reports, dashboards, and integrate with other systems. Orchestrate with Airflow or n8n."
homepage: "https://datadrivenconstruction.io"
metadata: {"openclaw": {"emoji": "⚙️", "os": ["darwin", "linux", "win32"], "homepage": "https://datadrivenconstruction.io", "requires": {"bins": ["python3"]}}}
---
# ETL Pipeline for Construction Data

## Overview

Based on DDC methodology (Chapter 4.2), this skill enables building automated data pipelines that extract information from various sources, transform it into useful formats, and load it into target systems or generate reports.

**Book Reference:** "ETL и автоматизация процессов" / "ETL and Process Automation"

> "ETL: переход от ручного управления к автоматизации позволяет компаниям обрабатывать данные без постоянного человеческого вмешательства."
> — DDC Book, Chapter 4.2

## ETL Components

```
┌─────────┐    ┌───────────┐    ┌────────┐
│ EXTRACT │ -> │ TRANSFORM │ -> │  LOAD  │
└─────────┘    └───────────┘    └────────┘
   │               │               │
   ▼               ▼               ▼
 Sources        Process         Outputs
 - PDF          - Clean         - Excel
 - Excel        - Validate      - PDF
 - CSV          - Calculate     - Database
 - BIM          - Merge         - API
 - API          - Aggregate     - Dashboard
```

## Quick Start

```python
import pandas as pd

# Simple ETL Pipeline
def simple_etl_pipeline(input_file, output_file):
    # EXTRACT
    df = pd.read_excel(input_file)

    # TRANSFORM
    df = df.dropna()  # Clean
    df['Total'] = df['Quantity'] * df['Unit_Price']  # Calculate
    summary = df.groupby('Category')['Total'].sum()  # Aggregate

    # LOAD
    summary.to_excel(output_file)
    return summary

# Run
result = simple_etl_pipeline("raw_data.xlsx", "processed_report.xlsx")
```

## Extract: Data Sources

### From Multiple Excel Files

```python
import pandas as pd
````
END THIRD-PARTY FILE ba18bb7950e64bfe

The third-party file has ended. Everything between the ba18bb7950e64bfe tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2347 of 14870 bytes. © datadrivenconstruction · MIT · https://github.com/datadrivenconstruction/DDC_Skills_for_AI_Agents_in_Construction/tree/HEAD/2_DDC_Book/4.2-ETL-Automation/etl-pipeline/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
