Skip to main content
skill CX Codex

data-engineering · travisjneuman — Claude Code skill: features and installation

The data-engineering Claude Code skill covers data pipelines and warehouse schema design. It activates for ETL/ELT work, stream processing, and dbt transformations.

MIT 105 installs ★ 114 updated 2026-10-25

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add travisjneuman/.claude --skill data-engineering -a claude-code
Open on GitHub

What data-engineering does

The skill explains ETL and ELT pipeline design, task orchestration with Airflow or Dagster, and data transformations in dbt. It also covers Kafka stream processing and warehouse schemas, including star schemas and slowly changing dimensions.

Its guidance includes data quality checks with Great Expectations and dbt tests, plus idempotent pipelines, incremental models, and dead letter queues. The Kafka examples expect a running broker reachable at localhost:9092.

The skill applies to pipeline design, warehouse schema work, and real-time data processing. Its code examples use Python and SQL.

What it helps with

  • design an ETL or ELT pipeline
  • organize warehouse data into facts and dimensions
  • process events through Kafka
  • add data quality tests in dbt
  • plan pipeline reruns and historical backfills

Example prompts for the agent

  • Design an ETL pipeline that loads data into a warehouse.
  • Propose a fact and dimension schema for orders and customers.
  • Set up Kafka event processing and data quality checks in dbt.

How to install data-engineering

1Skills CLI (Vercel)

npx skills add travisjneuman/.claude --skill data-engineering -a claude-code

2Manually: put the files in place

~/.claude/skills/data-engineering/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/data-engineering/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about data-engineering

How do I install data-engineering in Claude Code?

Run npx skills add travisjneuman/.claude --skill data-engineering -a claude-code. The skill file is placed at ~/.claude/skills/data-engineering/SKILL.md.

What does data-engineering do?

It guides data pipeline development, warehouse schema design, and stream processing. Its coverage includes Airflow, Dagster, dbt, and Kafka.

How does ETL differ from ELT in data-engineering?

ETL transforms data before loading it, while ELT loads raw data first and transforms it inside the warehouse. The skill outlines tools and use cases for both patterns.

Source text

SKILL.md · MIT
---
name: data-engineering
description: ETL/ELT pipelines, data warehousing (BigQuery, Snowflake, Redshift), stream processing (Kafka, Spark Streaming), orchestration (Airflow, Dagster, Prefect), dbt transformations, and data lake architecture. Use when building data pipelines, designing warehouse schemas, or implementing real-time data processing.
---

# Data Engineering

## Pipeline Architecture

### ETL vs ELT
| Pattern | When to Use | Tools |
|---------|-------------|-------|
| **ETL** | Transform before loading, data quality critical | Airflow + custom, Spark |
| **ELT** | Raw → warehouse → transform in-place | Fivetran + dbt, Airbyte + dbt |

### Orchestration

**Apache Airflow:**
```python
from airflow.decorators import dag, task
from datetime import datetime

@dag(schedule="@daily", start_date=datetime(2024, 1, 1), catchup=False)
def my_pipeline():
    @task()
    def extract() -> dict:
        return {"data": "extracted"}

    @task()
    def transform(data: dict) -> dict:
        return {"transformed": True}

    @task()
    def load(data: dict):
        # Load to warehouse
        pass

    raw = extract()
    transformed = transform(raw)
    load(transformed)

my_pipeline()
```

**Dagster (recommended for new projects):**
```python
from dagster import asset, Definitions

@asset
def raw_users():
    return extract_from_source()

@asset
def cleaned_users(raw_users):
    return clean_and_validate(raw_users)
```

## dbt Transformations

```sql
-- models/marts/dim_customers.sql
{{ config(materialized='table', schema='marts') }}

WITH source AS (
    SELECT * FROM {{ ref('stg_customers') }}
),
orders AS (
    SELECT customer_id, COUNT(*) as order_count, SUM(amount) as total_spent
    FROM {{ ref('stg_orders') }}
    GROUP BY customer_id
)
SELECT
    s.customer_id,
    s.name,
    s.email,
    COALESCE(o.order_count, 0) as lifetime_orders,
    COALESCE(o.total_spent, 0) as lifetime_value
FROM source s
LEFT JOIN orders o ON s.customer_id = o.customer_id
```

## Stream Processing

**Apache Kafka:**
```python
from confluent_kafka import Producer, Consumer

# Producer
producer = Producer({'bootstrap.servers': 'localhost:9092'})
producer.produce('events', key='user_123', value=json.dumps(event))
producer.flush()

# Consumer
consumer = Consumer({
    'bootstrap.servers': 'localhost:9092',
    'group.id': 'my-group',
    'auto.offset.reset': 'earliest'
})
consumer.subscribe(['events'])
```

## Data Warehouse Schema Design

### Star Schema
- **Fact tables:** Measurable events (orders, clicks, transactions)
- **Dimension tables:** Descriptive context (customers, products, dates)
- **Slowly Changing Dimensions:** Type 1 (overwrite), Type 2 (versioned rows), Type 3 (previous column)

### Data Quality
- **Great Expectations:** Schema validation, statistical tests, custom expectations
- **dbt tests:** `not_null`, `unique`, `accepted_values`, `relationships`, custom SQL tests
- **Data contracts:** Schema evolution policies, backward compatibility requirements

## Key Patterns
- **Idempotent pipelines:** Same input always produces same output, safe to rerun
- **Incremental models:** Process only new/changed data, use `updated_at` watermarks
- **Dead letter queues:** Route failed records for inspection without blocking pipeline
- **Backfill strategy:** Time-partitioned tables enable targeted historical reprocessing

© travisjneuman · MIT · travisjneuman/.claude

Similar skills

All skills for Data and databases →

data-engineering by other authors comes first.

data-engineering rohitg00 The data-engineering skill offers patterns for ETL pipelines, data warehouses, and Apache Spark jobs. Use it to shape data quality checks and warehouse schemas. CX Codex +1 ★ 2,858repo stars altimate-data-engineering-skills reason-machines altimate-data-engineering-skills is a Claude Code skill for data teams using dbt and Snowflake, guiding model building, build-error diagnosis, and warehouse query tuning. CX Codex +1 2,100installs Data Pipeline taste f Data Pipeline taste — a prompt for ChatGPT and Claude.ai establishes data engineering rules for reliable pipeline design. Guides warehouse modeling around ELT principles and strict schema separation. GP ChatGPT ★ 178,201repo stars zomato-ai-data-engineering-pipeline reason-machines zomato-ai-data-engineering-pipeline is a Claude Code skill for building a food delivery batch pipeline across S3, Snowflake, dbt, and Airflow, with OpenAI for review enrichment and text-to-SQL. CX Codex +1 128installs data-engineer davila7 data-engineer is a Claude Code skill for planning batch and streaming data pipelines. It joins work on warehouses, data quality, and orchestration. CX Codex +1 152installs senior-data-engineer Alireza Rezvani senior-data-engineer is a Claude Code skill for designing data pipelines and ETL/ELT systems. It activates for architecture choices, orchestration, data quality checks, and troubleshooting. CX Codex +1 1,109installs discover-data rand discover-data is a Claude Code skill that loads data pipeline and ETL guides. It triggers on tasks with Airflow, stream processing, batch pipelines, and data validation. CX Codex +1 101installs

From the same set117

The whole set →
serverless-development serverless-development — a skill for AI coding on serverless compute: AWS Lambda, Vercel Edge, Cloudflare Workers. Useful when an API or webhook has to scale on its own. CX Codex +1 133installs growth-engineering growth-engineering is a Claude Code skill with architecture patterns for feature flags, A/B testing, PLG funnels, and referral tracking. CX Codex +1 133installs email-systems email-systems — a skill for Claude Code that guides transactional email setup, template design, and deliverability records. Runs when configuring sending infrastructure, building HTML templates, or troubleshooting inboxing. CX Codex +1 132installs graphql-expert graphql-expert — a skill for Claude Code guides GraphQL schema design, resolver implementation, and query optimization. Runs when drafting schemas or resolving N+1 bottlenecks. CX Codex +1 132installs ar-vr-xr ar-vr-xr is a Claude Code skill that guides AR, VR, and XR app design for Unity, WebXR, Apple, and Meta, including spatial interactions, interfaces, and performance. CX Codex +1 125installs blockchain-web3 The blockchain-web3 skill supports Solidity contracts, dApps, DeFi, NFTs, and EVM work. Use it when you need contract security checks or Web3 wallet integration. CX Codex +1 109installs compliance-engineering The compliance-engineering Claude Code skill covers implementing SOC 2, HIPAA, GDPR, PCI-DSS, and FedRAMP requirements in code. It comes into play when adding audit logs, encryption, or access controls to a system. CX Codex +1 102installs game-development This Claude Code skill provides game-development guidance for Unity, Unreal Engine, and Godot. It activates for work on game mechanics, physics, AI, or engine-based projects. CX Codex +1 1installs

More in the catalog

Project sponsors

Companies that keep this analytics open