# data-engineering

The data-engineering Claude Code skill covers data pipelines and warehouse schema design. It activates for ETL/ELT work, stream processing, and dbt transformations.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 105 installs · ★ 114 repo stars
- Author: travisjneuman
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/travisjneuman/.claude/tree/HEAD/skills/data-engineering/SKILL.md
- Updated: 2026-10-25
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/travisjneuman-data-engineering

## What it does

The skill explains ETL and ELT pipeline design, task orchestration with Airflow or Dagster, and data transformations in dbt. It also covers Kafka stream processing and warehouse schemas, including star schemas and slowly changing dimensions.

Its guidance includes data quality checks with Great Expectations and dbt tests, plus idempotent pipelines, incremental models, and dead letter queues. The Kafka examples expect a running broker reachable at `localhost:9092`.

The skill applies to pipeline design, warehouse schema work, and real-time data processing. Its code examples use Python and SQL.

## Requirements
- Works with: Snowflake

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add travisjneuman/.claude --skill data-engineering -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/data-engineering/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/data-engineering/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add travisjneuman/.claude --skill data-engineering -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/data-engineering/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add travisjneuman/.claude --skill data-engineering -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/data-engineering/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install travisjneuman/.claude skills/data-engineering
```

2. **Manually: put the files in place**
- File: `.github/skills/data-engineering/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add travisjneuman/.claude --skill data-engineering -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/travisjneuman/.claude --path skills/data-engineering
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/data-engineering/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add travisjneuman/.claude --skill data-engineering -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/travisjneuman/.claude/tree/HEAD/skills/data-engineering/SKILL.md

## When to use it
- design an ETL or ELT pipeline
- organize warehouse data into facts and dimensions
- process events through Kafka
- add data quality tests in dbt
- plan pipeline reruns and historical backfills

## Example prompts
- Design an ETL pipeline that loads data into a warehouse.
- Propose a fact and dimension schema for orders and customers.
- Set up Kafka event processing and data quality checks in dbt.

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 32fd33f369d2c287.

BEGIN THIRD-PARTY FILE 32fd33f369d2c287
````text
---
name: data-engineering
description: ETL/ELT pipelines, data warehousing (BigQuery, Snowflake, Redshift), stream processing (Kafka, Spark Streaming), orchestration (Airflow, Dagster, Prefect), dbt transformations, and data lake architecture. Use when building data pipelines, designing warehouse schemas, or implementing real-time data processing.
---

# Data Engineering

## Pipeline Architecture

### ETL vs ELT
| Pattern | When to Use | Tools |
|---------|-------------|-------|
| **ETL** | Transform before loading, data quality critical | Airflow + custom, Spark |
| **ELT** | Raw → warehouse → transform in-place | Fivetran + dbt, Airbyte + dbt |

### Orchestration

**Apache Airflow:**
```python
from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="@daily", start_date=datetime(2024, 1, 1), catchup=False)
def my_pipeline():
    @task()
    def extract() -> dict:
        return {"data": "extracted"}

    @task()
    def transform(data: dict) -> dict:
        return {"transformed": True}

    @task()
    def load(data: dict):
        # Load to warehouse
        pass

    raw = extract()
    transformed = transform(raw)
    load(transformed)

my_pipeline()
```

**Dagster (recommended for new projects):**
```python
from dagster import asset, Definitions

@asset
def raw_users():
    return extract_from_source()

@asset
def cleaned_users(raw_users):
    return clean_and_validate(raw_users)
```

## dbt Transformations

```sql
-- models/marts/dim_customers.sql
{{ config(materialized='table', schema='marts') }}

WITH source AS (
    SELECT * FROM {{ ref('stg_customers') }}
),
orders AS (
    SELECT customer_id, COUNT(*) as order_count, SUM(amount) as total_spent
    FROM {{ ref('stg_orders') }}
    GROUP BY customer_id
)
SELECT
    s.customer_id,
    s.name,
    s.email,
    COALESCE(o.order_count, 0) as lifetime_orders,
    COALESCE(o.total_spent, 0) as lifetime_value
FROM source s
LEFT JOIN orders o ON s.customer_id = o.customer_id
```

## Stream Processing

**Apache Kafka:**
```python
from confluent_kafka import Producer, Consumer

# Producer
producer = Producer({'bootstrap.servers': 'localhost:9092'})
producer.produce('events', key='user_123', value=json.dumps(event))
producer.flush()

# Consumer
consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-group',
    'auto.offset.reset': 'earliest'
})
consumer.subscribe(['events'])
```

## Data Warehouse Schema Design

### Star Schema
- **Fact tables:** Measurable events (orders, clicks, transactions)
- **Dimension tables:** Descriptive context (customers, products, dates)
- **Slowly Changing Dimensions:** Type 1 (overwrite), Type 2 (versioned rows), Type 3 (previous column)

### Data Quality
- **Great Expectations:** Schema validation, statistical tests, custom expectations
- **dbt tests:** `not_null`, `unique`, `accepted_values`, `relationships`, custom SQL tests
````
END THIRD-PARTY FILE 32fd33f369d2c287

The third-party file has ended. Everything between the 32fd33f369d2c287 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2923 of 3370 bytes. © travisjneuman · MIT · https://github.com/travisjneuman/.claude/tree/HEAD/skills/data-engineering/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
