# data-engineering-medallion-pipeline

This skill helps agents build ELT pipelines with MinIO, Airbyte, PostgreSQL, DBT, and Airflow. Use it for Bronze, Silver, and Gold data layers.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 2000 installs · ★ 5
- Author: reason-machines
- License: LicenseRef-Unknown · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/data-skills/tree/HEAD/skills/data-engineering-medallion-pipeline/SKILL.md
- Updated: 2026-10-09
- HTML: https://affpapa.org/en/nevibecode/skills/data-engineering-medallion-pipeline

## What it does

The skill guides an end-to-end pipeline that takes source data from MinIO through Airbyte, loads it into PostgreSQL, and transforms it in DBT. Bronze models extract fields from JSONB, Silver models clean and type records while removing duplicates, and Gold models produce business aggregates. Airflow coordinates the stages through DAG files.

The project also includes DBT quality tests, SCD Type 2 snapshots, scripts for sending CSV and JSON files to MinIO, and connection checks. Prometheus and Grafana cover infrastructure monitoring, while Power BI can consume business metrics. The local stack requires Docker 20.10+, Docker Compose 2.0+, and at least 8GB RAM; without them, the services will not start.

## Requirements
- Works with: PostgreSQL

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-medallion-pipeline -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/data-engineering-medallion-pipeline/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/data-engineering-medallion-pipeline/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-medallion-pipeline -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/data-engineering-medallion-pipeline/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-medallion-pipeline -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/data-engineering-medallion-pipeline/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/data-skills data-engineering-medallion-pipeline
```

2. **Manually: put the files in place**
- File: `.github/skills/data-engineering-medallion-pipeline/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-medallion-pipeline -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/data-skills --path skills/data-engineering-medallion-pipeline
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/data-engineering-medallion-pipeline/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-medallion-pipeline -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/data-skills/tree/HEAD/skills/data-engineering-medallion-pipeline/SKILL.md

## When to use it
- build an ELT pipeline with medallion layers
- load source data from MinIO through Airbyte
- clean and type records in DBT models
- run automated data quality checks
- orchestrate stages with Airflow DAGs

## Example prompts
- Build a local ELT pipeline with Bronze, Silver, and Gold layers using MinIO, Airbyte, PostgreSQL, DBT, and Airflow.
- Create DBT models that clean, type, and deduplicate records in the Silver layer.
- Add an Airflow DAG that loads data from MinIO and runs the quality checks.

## Similar
- [postgresql-database-engineering](https://affpapa.org/en/nevibecode/skills/postgresql-database-engineering.md) — skill · 1800 installs
- [firebase-data-connect](https://affpapa.org/en/nevibecode/skills/firebase-data-connect.md) — skill · 168800 installs
- [connecting-to-data-source](https://affpapa.org/en/nevibecode/skills/connecting-to-data-source.md) — skill · 6800 installs
- [ingesting-into-data-lake](https://affpapa.org/en/nevibecode/skills/ingesting-into-data-lake.md) — skill · 6600 installs
- [data-context-extractor](https://affpapa.org/en/nevibecode/skills/data-context-extractor.md) — skill · 3100 installs
- [data-model-creation](https://affpapa.org/en/nevibecode/skills/data-model-creation.md) — skill · 1000 installs

From the same set: [reason-machines/data-skills](https://affpapa.org/en/nevibecode/by/reason-machines/data-skills.md) — 74 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md