data-engineering · travisjneuman — Claude Code skill: features and installation
The data-engineering Claude Code skill covers data pipelines and warehouse schema design. It activates for ETL/ELT work, stream processing, and dbt transformations.
MIT 105 installs ★ 114 updated 2026-10-25
npx skills add travisjneuman/.claude --skill data-engineering -a claude-codeWhat data-engineering does
The skill explains ETL and ELT pipeline design, task orchestration with Airflow or Dagster, and data transformations in dbt. It also covers Kafka stream processing and warehouse schemas, including star schemas and slowly changing dimensions.
Its guidance includes data quality checks with Great Expectations and dbt tests, plus idempotent pipelines, incremental models, and dead letter queues. The Kafka examples expect a running broker reachable at localhost:9092.
The skill applies to pipeline design, warehouse schema work, and real-time data processing. Its code examples use Python and SQL.
What it helps with
- design an ETL or ELT pipeline
- organize warehouse data into facts and dimensions
- process events through Kafka
- add data quality tests in dbt
- plan pipeline reruns and historical backfills
Example prompts for the agent
- Design an ETL pipeline that loads data into a warehouse.
- Propose a fact and dimension schema for orders and customers.
- Set up Kafka event processing and data quality checks in dbt.
How to install data-engineering
1Skills CLI (Vercel)
npx skills add travisjneuman/.claude --skill data-engineering -a claude-code2Manually: put the files in place
~/.claude/skills/data-engineering/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/data-engineering/
1Skills CLI (Vercel)
npx skills add travisjneuman/.claude --skill data-engineering -a codex2Manually: put the files in place
.agents/skills/data-engineering/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add travisjneuman/.claude --skill data-engineering -a cursor2Manually: put the files in place
.cursor/skills/data-engineering/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install travisjneuman/.claude skills/data-engineering2Manually: put the files in place
.github/skills/data-engineering/SKILL.md1Skills CLI (Vercel)
npx skills add travisjneuman/.claude --skill data-engineering -a gemini-cli2Platform CLI
gemini skills install https://github.com/travisjneuman/.claude --path skills/data-engineeringManually: put the files in place
.devin/skills/data-engineering/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add travisjneuman/.claude --skill data-engineering -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about data-engineering
How do I install data-engineering in Claude Code?
Run npx skills add travisjneuman/.claude --skill data-engineering -a claude-code. The skill file is placed at ~/.claude/skills/data-engineering/SKILL.md.
What does data-engineering do?
It guides data pipeline development, warehouse schema design, and stream processing. Its coverage includes Airflow, Dagster, dbt, and Kafka.
How does ETL differ from ELT in data-engineering?
ETL transforms data before loading it, while ELT loads raw data first and transforms it inside the warehouse. The skill outlines tools and use cases for both patterns.
Source text
SKILL.md · MIT
---
name: data-engineering
description: ETL/ELT pipelines, data warehousing (BigQuery, Snowflake, Redshift), stream processing (Kafka, Spark Streaming), orchestration (Airflow, Dagster, Prefect), dbt transformations, and data lake architecture. Use when building data pipelines, designing warehouse schemas, or implementing real-time data processing.
---
# Data Engineering
## Pipeline Architecture
### ETL vs ELT
| Pattern | When to Use | Tools |
|---------|-------------|-------|
| **ETL** | Transform before loading, data quality critical | Airflow + custom, Spark |
| **ELT** | Raw → warehouse → transform in-place | Fivetran + dbt, Airbyte + dbt |
### Orchestration
**Apache Airflow:**
```python
from airflow.decorators import dag, task
from datetime import datetime
@dag(schedule="@daily", start_date=datetime(2024, 1, 1), catchup=False)
def my_pipeline():
@task()
def extract() -> dict:
return {"data": "extracted"}
@task()
def transform(data: dict) -> dict:
return {"transformed": True}
@task()
def load(data: dict):
# Load to warehouse
pass
raw = extract()
transformed = transform(raw)
load(transformed)
my_pipeline()
```
**Dagster (recommended for new projects):**
```python
from dagster import asset, Definitions
@asset
def raw_users():
return extract_from_source()
@asset
def cleaned_users(raw_users):
return clean_and_validate(raw_users)
```
## dbt Transformations
```sql
-- models/marts/dim_customers.sql
{{ config(materialized='table', schema='marts') }}
WITH source AS (
SELECT * FROM {{ ref('stg_customers') }}
),
orders AS (
SELECT customer_id, COUNT(*) as order_count, SUM(amount) as total_spent
FROM {{ ref('stg_orders') }}
GROUP BY customer_id
)
SELECT
s.customer_id,
s.name,
s.email,
COALESCE(o.order_count, 0) as lifetime_orders,
COALESCE(o.total_spent, 0) as lifetime_value
FROM source s
LEFT JOIN orders o ON s.customer_id = o.customer_id
```
## Stream Processing
**Apache Kafka:**
```python
from confluent_kafka import Producer, Consumer
# Producer
producer = Producer({'bootstrap.servers': 'localhost:9092'})
producer.produce('events', key='user_123', value=json.dumps(event))
producer.flush()
# Consumer
consumer = Consumer({
'bootstrap.servers': 'localhost:9092',
'group.id': 'my-group',
'auto.offset.reset': 'earliest'
})
consumer.subscribe(['events'])
```
## Data Warehouse Schema Design
### Star Schema
- **Fact tables:** Measurable events (orders, clicks, transactions)
- **Dimension tables:** Descriptive context (customers, products, dates)
- **Slowly Changing Dimensions:** Type 1 (overwrite), Type 2 (versioned rows), Type 3 (previous column)
### Data Quality
- **Great Expectations:** Schema validation, statistical tests, custom expectations
- **dbt tests:** `not_null`, `unique`, `accepted_values`, `relationships`, custom SQL tests
- **Data contracts:** Schema evolution policies, backward compatibility requirements
## Key Patterns
- **Idempotent pipelines:** Same input always produces same output, safe to rerun
- **Incremental models:** Process only new/changed data, use `updated_at` watermarks
- **Dead letter queues:** Route failed records for inspection without blocking pipeline
- **Backfill strategy:** Time-partitioned tables enable targeted historical reprocessing
© travisjneuman · MIT · travisjneuman/.claude
Similar skills
All skills for Data and databases →data-engineering by other authors comes first.