Skip to main content
skill CX Codex

data-pipeline-engineer — Claude Code skill: setup and features

data-pipeline-engineer guides data pipeline design, Medallion storage layers, Airflow DAG orchestration, and dbt models. Activates for batch processing and streaming workloads.

MIT 215 installs ★ 274 updated 2026-09-06

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a claude-code
Open on GitHub

What data-pipeline-engineer does

This skill guides ETL and ELT development. It structures storage into Bronze, Silver, and Gold layers, configures stream processing via Kafka and Spark, and writes dimensional models in dbt. Data quality gates use Great Expectations suites alongside dbt test blocks.

Included reference files provide Airflow DAG layouts, Kafka-to-Delta streaming code, and a validation script at ./scripts/validate-pipeline.sh. The script checks dbt conventions and Spark settings. Terminal execution is locked down to dbt, spark-submit, airflow, and python commands.

It does not handle API design, machine learning training, or BI dashboarding. When pipeline data feeds an API, pair this tool with api-architect.

What it helps with

  • Build Medallion architecture layers from raw ingestion to Gold marts
  • Write production Airflow DAGs with sensors and exponential retry backoffs
  • Implement streaming pipelines from Kafka to Delta Lake using Spark
  • Configure automated quality suites with Great Expectations and dbt

Example prompts for the agent

  • Design a dbt project structure separating staging, intermediate, and marts
  • Generate an Airflow DAG with task groups, sensors, and retry configurations
  • Add watermarking to a Spark Streaming processor for late-arriving events

How to install data-pipeline-engineer

1Skills CLI (Vercel)

npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a claude-code

2Manually: put the files in place

~/.claude/skills/data-pipeline-engineer/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/data-pipeline-engineer/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about data-pipeline-engineer

How do I install data-pipeline-engineer in Claude Code?

Run npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a claude-code. The file is saved to ~/.claude/skills/data-pipeline-engineer/SKILL.md.

What does data-pipeline-engineer do?

It drafts architecture patterns and writes pipeline code for dbt, Airflow, Spark, and Kafka workloads.

Can data-pipeline-engineer build web APIs or train ML models?

No. The skill explicitly excludes API design, machine learning model training, and dashboard creation, focusing purely on data engineering.

Source text

SKILL.md · MIT
---
name: data-pipeline-engineer
description: 'Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming
  data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).'
allowed-tools: Read,Write,Edit,Bash(dbt:*,spark-submit:*,airflow:*,python:*)
metadata:
  category: Data & Analytics
  pairs-with:
  - skill: api-architect
    reason: APIs that consume pipeline data
  - skill: devops-automator
    reason: Orchestrate pipeline infrastructure
  tags:
  - etl
  - spark
  - kafka
  - airflow
  - data-warehouse
---

# Data Pipeline Engineer

Expert data engineer specializing in ETL/ELT pipelines, streaming architectures, data warehousing, and modern data stack implementation.

## Quick Start

1. **Identify sources** - data formats, volumes, freshness requirements
2. **Choose architecture** - Medallion (Bronze/Silver/Gold), Lambda, or Kappa
3. **Design layers** - staging → intermediate → marts (dbt pattern)
4. **Add quality gates** - Great Expectations or dbt tests at each layer
5. **Orchestrate** - Airflow DAGs with sensors and retries
6. **Monitor** - lineage, freshness, anomaly detection

## Core Capabilities

| Capability | Technologies | Key Patterns |
|------------|--------------|--------------|
| **Batch Processing** | Spark, dbt, Databricks | Incremental, partitioning, Delta/Iceberg |
| **Stream Processing** | Kafka, Flink, Spark Streaming | Watermarks, exactly-once, windowing |
| **Orchestration** | Airflow, Dagster, Prefect | DAG design, sensors, task groups |
| **Data Modeling** | dbt, SQL | Kimball, Data Vault, SCD |
| **Data Quality** | Great Expectations, dbt tests | Validation suites, freshness |

## Architecture Patterns

### Medallion Architecture (Recommended)
```
BRONZE (Raw)     → Exact source copy, schema-on-read, partitioned by ingestion
      ↓ Cleaning, Deduplication
SILVER (Cleansed) → Validated, standardized, business logic applied
      ↓ Aggregation, Enrichment
GOLD (Business)   → Dimensional models, aggregates, ready for BI/ML
```

### Lambda vs Kappa
- **Lambda**: Batch + Stream layers → merged serving layer (complex but complete)
- **Kappa**: Stream-only with replay → simpler but requires robust streaming

## Reference Examples

Full implementation examples in `./references/`:

| File | Description |
|------|-------------|
| `dbt-project-structure.md` | Complete dbt layout with staging, intermediate, marts |
| `airflow-dag.py` | Production DAG with sensors, task groups, quality checks |
| `spark-streaming.py` | Kafka-to-Delta processor with windowing |
| `great-expectations-suite.json` | Comprehensive data quality expectation suite |

## Anti-Patterns (10 Critical Mistakes)

### 1. Full Table Refreshes
**Symptom**: Truncate and rebuild entire tables every run
**Fix**: Use incremental models with `is_incremental()`, partition by date

### 2. Tight Coupling to Source Schemas
**Symptom**: Pipeline breaks when upstream adds/removes columns
**Fix**: Explicit source contracts, select only needed columns in staging

### 3. Monolithic DAGs
**Symptom**: One 200-task DAG running 8 hours
**Fix**: Domain-specific DAGs, ExternalTaskSensor for dependencies

### 4. No Data Quality Gates
**Symptom**: Bad data reaches production before detection
**Fix**: Great Expectations or dbt tests at each layer, block on failures

### 5. Processing Before Archiving
**Symptom**: Raw data transformed without preserving original
**Fix**: Always land raw in Bronze first, make transformations reproducible

### 6. Hardcoded Dates in Queries
**Symptom**: Manual updates needed for date filters
**Fix**: Use Airflow templating (e.g., `ds` variable) or dynamic date functions

### 7. Missing Watermarks in Streaming
**Symptom**: Unbounded state growth, OOM in long-running jobs
**Fix**: Add `withWatermark()` to handle late-arriving data

### 8. No Retry/Backoff Strategy
**Symptom**: Transient failures cause DAG failures
**Fix**: `retries=3`, `retry_exponential_backoff=True`, `max_retry_delay`

### 9. Undocumented Data Lineage
**Symptom**: No one knows where data comes from or who uses it
**Fix**: dbt docs, data catalog integration, column-level lineage

### 10. Testing Only in Production
**Symptom**: Bugs discovered by stakeholders, not engineers
**Fix**: dbt `--target dev`, sample datasets, CI/CD for models

## Quality Checklist

**Pipeline Design:**
- [ ] Incremental processing where possible
- [ ] Idempotent transformations (re-runnable safely)
- [ ] Partitioning strategy defined and documented
- [ ] Backfill procedures documented

**Data Quality:**
- [ ] Tests at Bronze layer (schema, nulls, ranges)
- [ ] Tests at Silver layer (business rules, referential integrity)
- [ ] Tests at Gold layer (aggregation checks, trend monitoring)
- [ ] Anomaly detection for volumes and distributions

**Orchestration:**
- [ ] Retry and alerting configured
- [ ] SLAs defined and monitored
- [ ] Cross-DAG dependencies use sensors
- [ ] max_active_runs prevents parallel conflicts

**Operations:**
- [ ] Data lineage documented
- [ ] Runbooks for common failures
- [ ] Monitoring dashboards for pipeline health
- [ ] On-call procedures defined

## Validation Script

Run `./scripts/validate-pipeline.sh` to check:
- dbt project structure and conventions
- Airflow DAG best practices
- Spark job configurations
- Data quality setup

## External Resources

- [dbt Best Practices](https://docs.getdbt.com/guides/best-practices)
- [Airflow Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)
- [Great Expectations Docs](https://docs.greatexpectations.io/)
- [Delta Lake Guide](https://docs.delta.io/latest/index.html)
- [Kafka Streams](https://kafka.apache.org/documentation/streams/)

© curiositech · MIT · curiositech/some_claude_skills

Similar skills

All skills for Data and databases →

From the same set180

The whole set →
prompt-engineer prompt-engineer — a skill for crafting and debugging prompts. Eliminates hallucinations, improves response consistency, and strengthens agent instructions through systematic optimization techniques. CX Codex +1 217installs team-builder team-builder designs team structures using organizational psychology. When a required role is missing from the skill library, it writes a new skill right away. CX Codex +1 216installs react-performance-optimizer react-performance-optimizer diagnoses React render bottlenecks to hit 60fps. It tackles sluggish lists, unnecessary component updates, and oversized bundles. CX Codex +1 215installs digital-estate-planner digital-estate-planner — a skill for organizing digital life ahead of death, incapacity, or emergencies. It inventories accounts, documents passwords and crypto, and writes instructions for a digital executor. CX Codex +1 215installs hipaa-compliance The hipaa-compliance skill guides PHI handling with audit logging, log sanitization, and session checks. Use it when coding features that touch health data. CX Codex +1 215installs web-wave-designer web-wave-designer — a skill that builds ocean waves, ripples, and underwater distortion for the web with SVG filters and CSS animation. Handy when a page needs a living water background. CX Codex +1 214installs test-automation-expert test-automation-expert — a test automation skill that covers strategy, framework setup, coverage gaps and flaky test debugging. Useful when a project needs tests from scratch or existing ones keep breaking. CX Codex +1 212installs very-long-text-summarization very-long-text-summarization — a Claude Code skill that digests books, handbooks and large codebases with hierarchical multi-pass extraction. It outputs a structured knowledge map, not a plain compressed retelling. CX Codex +1 210installs

Project sponsors

Companies that keep this analytics open