Skip to main content
skill CX Codex

data-engineering · rohitg00 — Claude Code skill for ETL and data quality

The data-engineering skill offers patterns for ETL pipelines, data warehouses, and Apache Spark jobs. Use it to shape data quality checks and warehouse schemas.

Apache-2.0 ★ 2,858 updated 2026-05-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add rohitg00/awesome-claude-code-toolkit --skill data-engineering -a claude-code
Open on GitHub

What data-engineering does

The examples cover an ETL flow with incremental extraction and batch upserts, an Apache Spark job that aggregates order data, and SQL-based quality checks. A star schema illustrates fact and dimension tables.

The skill flags pitfalls such as row-by-row processing, missing checks between stages, and full-table scans where incremental loads would work. Critical checks raise an error when they fail. The Spark example uses PySpark, while the pipeline calls fetch_all and upsert_batch, so the database connections need to provide those methods.

What it helps with

  • design an ETL flow with distinct extract, transform, and load stages
  • aggregate daily revenue and calculate a rolling average in Spark
  • check a warehouse table for null IDs, negative totals, and duplicate orders
  • draft a star schema with fact and dimension tables

Example prompts for the agent

  • Design an incremental order ETL pipeline with data quality checks.
  • Show a Spark job that groups daily revenue by product category and calculates a rolling average.
  • Draft a star schema for orders and add checks for duplicate order IDs.

How to install data-engineering

1Skills CLI (Vercel)

npx skills add rohitg00/awesome-claude-code-toolkit --skill data-engineering -a claude-code

2Manually: put the files in place

~/.claude/skills/data-engineering/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/data-engineering/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about data-engineering

How do I install data-engineering in Claude Code?

Run npx skills add rohitg00/awesome-claude-code-toolkit --skill data-engineering -a claude-code. The skill file is at ~/.claude/skills/data-engineering/SKILL.md.

What does the data-engineering skill do?

It provides examples for ETL pipelines, Apache Spark processing, data quality checks, and warehouse schemas. It also outlines pipeline anti-patterns and loading practices.

What does the Apache Spark example require?

It uses PySpark and reads Parquet data from s3://data-lake/orders/ and s3://data-lake/customers/, then writes results to s3://data-warehouse/daily_revenue/.

Source text

SKILL.md · Apache-2.0
---
name: data-engineering
description: Data engineering patterns for ETL pipelines, data warehousing, Apache Spark, and data quality validation
---

# Data Engineering

## ETL Pipeline Pattern

```python
from datetime import datetime
from dataclasses import dataclass

@dataclass
class PipelineResult:
    records_extracted: int
    records_transformed: int
    records_loaded: int
    errors: list[str]
    duration_seconds: float

class OrderPipeline:
    def __init__(self, source_db, warehouse_db):
        self.source = source_db
        self.warehouse = warehouse_db

    def extract(self, since: datetime) -> list[dict]:
        query = """
            SELECT o.*, c.name as customer_name, c.segment
            FROM orders o
            JOIN customers c ON o.customer_id = c.id
            WHERE o.updated_at > %s
        """
        return self.source.fetch_all(query, [since])

    def transform(self, records: list[dict]) -> list[dict]:
        transformed = []
        for record in records:
            transformed.append({
                "order_id": record["id"],
                "customer_name": record["customer_name"],
                "segment": record["segment"].upper(),
                "total_amount": float(record["total"]),
                "order_date": record["created_at"].date(),
                "fiscal_quarter": get_fiscal_quarter(record["created_at"]),
                "is_high_value": float(record["total"]) > 1000,
                "loaded_at": datetime.utcnow(),
            })
        return transformed

    def load(self, records: list[dict]) -> int:
        return self.warehouse.upsert_batch(
            table="fact_orders",
            records=records,
            conflict_keys=["order_id"],
            batch_size=5000,
        )

    def run(self, since: datetime) -> PipelineResult:
        start = datetime.utcnow()
        raw = self.extract(since)
        clean = self.transform(raw)
        loaded = self.load(clean)
        return PipelineResult(
            records_extracted=len(raw),
            records_transformed=len(clean),
            records_loaded=loaded,
            errors=[],
            duration_seconds=(datetime.utcnow() - start).total_seconds(),
        )
```

## Apache Spark Processing

```python
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
from pyspark.sql.window import Window

spark = SparkSession.builder \
    .appName("sales-analytics") \
    .config("spark.sql.adaptive.enabled", "true") \
    .config("spark.sql.shuffle.partitions", "200") \
    .getOrCreate()

orders = spark.read.parquet("s3://data-lake/orders/")
customers = spark.read.parquet("s3://data-lake/customers/")

daily_revenue = (
    orders
    .filter(F.col("status") == "completed")
    .withColumn("order_date", F.to_date("created_at"))
    .groupBy("order_date", "product_category")
    .agg(
        F.sum("total_amount").alias("revenue"),
        F.count("id").alias("order_count"),
        F.avg("total_amount").alias("avg_order_value"),
    )
    .withColumn(
        "revenue_7d_avg",
        F.avg("revenue").over(
            Window.partitionBy("product_category")
            .orderBy("order_date")
            .rowsBetween(-6, 0)
        )
    )
)

daily_revenue.write \
    .partitionBy("order_date") \
    .mode("overwrite") \
    .parquet("s3://data-warehouse/daily_revenue/")
```

## Data Quality Checks

```python
from dataclasses import dataclass

@dataclass
class QualityCheck:
    name: str
    query: str
    threshold: float
    severity: str

CHECKS = [
    QualityCheck(
        name="null_customer_ids",
        query="SELECT COUNT(*) FROM fact_orders WHERE customer_id IS NULL",
        threshold=0,
        severity="critical",
    ),
    QualityCheck(
        name="negative_amounts",
        query="SELECT COUNT(*) FROM fact_orders WHERE total_amount < 0",
        threshold=0,
        severity="critical",
    ),
    QualityCheck(
        name="duplicate_orders",
        query="SELECT COUNT(*) - COUNT(DISTINCT order_id) FROM fact_orders",
        threshold=0,
        severity="warning",
    ),
    QualityCheck(
        name="freshness",
        query="SELECT EXTRACT(EPOCH FROM NOW() - MAX(loaded_at))/3600 FROM fact_orders",
        threshold=2.0,
        severity="warning",
    ),
]

def run_quality_checks(db, checks: list[QualityCheck]) -> list[dict]:
    results = []
    for check in checks:
        value = db.fetch_scalar(check.query)
        passed = value <= check.threshold
        results.append({
            "name": check.name,
            "value": value,
            "threshold": check.threshold,
            "passed": passed,
            "severity": check.severity,
        })
        if not passed and check.severity == "critical":
            raise DataQualityError(f"Critical check failed: {check.name} = {value}")
    return results
```

## Data Warehouse Schema (Star Schema)

```sql
CREATE TABLE dim_customers (
    customer_key    BIGINT PRIMARY KEY,
    customer_id     VARCHAR(50) NOT NULL,
    name            VARCHAR(200),
    segment         VARCHAR(50),
    country         VARCHAR(100),
    valid_from      TIMESTAMP NOT NULL,
    valid_to        TIMESTAMP,
    is_current      BOOLEAN DEFAULT TRUE
);

CREATE TABLE dim_products (
    product_key     BIGINT PRIMARY KEY,
    product_id      VARCHAR(50) NOT NULL,
    name            VARCHAR(200),
    category        VARCHAR(100),
    subcategory     VARCHAR(100)
);

CREATE TABLE fact_orders (
    order_key       BIGINT PRIMARY KEY,
    order_id        VARCHAR(50) UNIQUE NOT NULL,
    customer_key    BIGINT REFERENCES dim_customers(customer_key),
    product_key     BIGINT REFERENCES dim_products(product_key),
    order_date_key  INT,
    quantity        INT,
    unit_price      DECIMAL(10,2),
    total_amount    DECIMAL(12,2),
    loaded_at       TIMESTAMP DEFAULT NOW()
);
```

## Anti-Patterns

- Processing data row-by-row instead of in batches or sets
- Not partitioning large tables by date or category
- Missing data quality checks between pipeline stages
- Loading raw data directly into the warehouse without transformation
- Using full table scans when incremental loads would suffice
- Not tracking data lineage (where data came from, when it was processed)

## Checklist

- [ ] Pipelines follow Extract-Transform-Load with clear stage separation
- [ ] Incremental processing based on watermarks or change data capture
- [ ] Data quality checks run after each pipeline stage
- [ ] Warehouse uses star or snowflake schema with dimension and fact tables
- [ ] Spark jobs use adaptive query execution and appropriate partitioning
- [ ] Idempotent loads (re-running produces the same result)
- [ ] Data freshness monitored with automated alerts
- [ ] Schema evolution handled gracefully (additive changes preferred)

© rohitg00 · Apache-2.0 · rohitg00/awesome-claude-code-toolkit

Similar skills

All skills for Data and databases →

data-engineering by other authors comes first.

data-engineering travisjneuman The data-engineering Claude Code skill covers data pipelines and warehouse schema design. It activates for ETL/ELT work, stream processing, and dbt transformations. CX Codex +1 105installs data-engineering-study-material reason-machines data-engineering-study-material — a study guide skill for data engineering: explains concepts, shows pipeline examples, and helps with interview prep. CX Codex +1 2,000installs data-engineer Seth Hobson data-engineer — a subagent for data pipelines, warehouses, and streaming architectures. Use it for Apache Spark, dbt, Airflow, and cloud data platform work. CX Codex 3,194installs enterprise-data-engineering-pipeline-ssis-pyspark reason-machines enterprise-data-engineering-pipeline-ssis-pyspark guides coding assistants through building an ELT pipeline with SQL Server star schemas, SSIS orchestration packages, and PySpark analytics jobs. CX Codex +1 2,000installs agentic-bundle-aas-data-engineering-platform sickn33 and contributors The agentic-bundle-aas-data-engineering-platform plugin for Claude Code bundles skills for data engineering tasks. Marketplace installation takes two commands. CX Codex ★ 50,109repo stars data-quality-frameworks Seth Hobson The data-quality-frameworks skill sets up data checks with Great Expectations, dbt tests, and data contracts. Use it for validation pipelines, CI/CD checks, and schema rules. CX Codex +1 11,230installs suggesting-data-imports PostHog The suggesting-data-imports skill suggests importing external data into PostHog when a query lacks tables for payments, subscriptions, CRM records, or bucket files. CX Codex +1 401installs

From the same set178

The whole set →
academic-researcher academic-researcher runs systematic literature reviews, inspects study methodology, and compiles evidence across papers. Built for technical and scientific research. CX Codex ★ 2,858repo stars devops-automation The devops-automation skill guides CI/CD design for GitHub Actions and configuration work with Docker, Kubernetes, Helm, and ArgoCD. Use it when planning an application build and deployment flow. CX Codex +1 ★ 2,858repo stars technology-scout technology-scout is a Claude Code subagent that evaluates emerging technologies, compares build-versus-buy options, and assesses vendors to inform adoption decisions. CX Codex ★ 2,858repo stars react-patterns react-patterns is a Claude Code skill with React 19 patterns for Server Components, Server Actions, Suspense, and state management. CX Codex +1 ★ 2,858repo stars python-best-practices python-best-practices guides Claude Code toward modern Python style: type hints, dataclasses, async code, project layout and pytest setup. CX Codex +1 ★ 2,858repo stars golang-idioms golang-idioms is a Claude Code skill that provides idiomatic Go guidance for error handling, concurrency, and testing. CX Codex +1 ★ 2,858repo stars workflow-optimizer The workflow-optimizer plugin analyzes development workflows and suggests ways to improve them. Use it when you want recommendations on how work in a project is organized. ★ 2,858repo stars web-dev web-dev — a plugin for Claude Code that automates full-stack web app creation: generates project structure, components, and pages in a single step. ★ 2,858repo stars

Project sponsors

Companies that keep this analytics open