data-pipeline-engineer — Claude Code skill: setup and features
data-pipeline-engineer guides data pipeline design, Medallion storage layers, Airflow DAG orchestration, and dbt models. Activates for batch processing and streaming workloads.
MIT 215 installs ★ 274 updated 2026-09-06
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a claude-codeWhat data-pipeline-engineer does
This skill guides ETL and ELT development. It structures storage into Bronze, Silver, and Gold layers, configures stream processing via Kafka and Spark, and writes dimensional models in dbt. Data quality gates use Great Expectations suites alongside dbt test blocks.
Included reference files provide Airflow DAG layouts, Kafka-to-Delta streaming code, and a validation script at ./scripts/validate-pipeline.sh. The script checks dbt conventions and Spark settings. Terminal execution is locked down to dbt, spark-submit, airflow, and python commands.
It does not handle API design, machine learning training, or BI dashboarding. When pipeline data feeds an API, pair this tool with api-architect.
What it helps with
- Build Medallion architecture layers from raw ingestion to Gold marts
- Write production Airflow DAGs with sensors and exponential retry backoffs
- Implement streaming pipelines from Kafka to Delta Lake using Spark
- Configure automated quality suites with Great Expectations and dbt
Example prompts for the agent
- Design a dbt project structure separating staging, intermediate, and marts
- Generate an Airflow DAG with task groups, sensors, and retry configurations
- Add watermarking to a Spark Streaming processor for late-arriving events
How to install data-pipeline-engineer
1Skills CLI (Vercel)
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a claude-code2Manually: put the files in place
~/.claude/skills/data-pipeline-engineer/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/data-pipeline-engineer/
1Skills CLI (Vercel)
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a codex2Manually: put the files in place
.agents/skills/data-pipeline-engineer/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a cursor2Manually: put the files in place
.cursor/skills/data-pipeline-engineer/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install curiositech/some_claude_skills .claude/skills/data-pipeline-engineer2Manually: put the files in place
.github/skills/data-pipeline-engineer/SKILL.md1Skills CLI (Vercel)
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a gemini-cli2Platform CLI
gemini skills install https://github.com/curiositech/some_claude_skills --path .claude/skills/data-pipeline-engineerManually: put the files in place
.devin/skills/data-pipeline-engineer/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about data-pipeline-engineer
How do I install data-pipeline-engineer in Claude Code?
Run npx skills add curiositech/some_claude_skills --skill data-pipeline-engineer -a claude-code. The file is saved to ~/.claude/skills/data-pipeline-engineer/SKILL.md.
What does data-pipeline-engineer do?
It drafts architecture patterns and writes pipeline code for dbt, Airflow, Spark, and Kafka workloads.
Can data-pipeline-engineer build web APIs or train ML models?
No. The skill explicitly excludes API design, machine learning model training, and dashboard creation, focusing purely on data engineering.
Source text
SKILL.md · MIT
---
name: data-pipeline-engineer
description: 'Expert data engineer for ETL/ELT pipelines, streaming, data warehousing. Activate on: data pipeline, ETL, ELT, data warehouse, Spark, Kafka, Airflow, dbt, data modeling, star schema, streaming
data, batch processing, data quality. NOT for: API design (use api-architect), ML training (use ML skills), dashboards (use design skills).'
allowed-tools: Read,Write,Edit,Bash(dbt:*,spark-submit:*,airflow:*,python:*)
metadata:
category: Data & Analytics
pairs-with:
- skill: api-architect
reason: APIs that consume pipeline data
- skill: devops-automator
reason: Orchestrate pipeline infrastructure
tags:
- etl
- spark
- kafka
- airflow
- data-warehouse
---
# Data Pipeline Engineer
Expert data engineer specializing in ETL/ELT pipelines, streaming architectures, data warehousing, and modern data stack implementation.
## Quick Start
1. **Identify sources** - data formats, volumes, freshness requirements
2. **Choose architecture** - Medallion (Bronze/Silver/Gold), Lambda, or Kappa
3. **Design layers** - staging → intermediate → marts (dbt pattern)
4. **Add quality gates** - Great Expectations or dbt tests at each layer
5. **Orchestrate** - Airflow DAGs with sensors and retries
6. **Monitor** - lineage, freshness, anomaly detection
## Core Capabilities
| Capability | Technologies | Key Patterns |
|------------|--------------|--------------|
| **Batch Processing** | Spark, dbt, Databricks | Incremental, partitioning, Delta/Iceberg |
| **Stream Processing** | Kafka, Flink, Spark Streaming | Watermarks, exactly-once, windowing |
| **Orchestration** | Airflow, Dagster, Prefect | DAG design, sensors, task groups |
| **Data Modeling** | dbt, SQL | Kimball, Data Vault, SCD |
| **Data Quality** | Great Expectations, dbt tests | Validation suites, freshness |
## Architecture Patterns
### Medallion Architecture (Recommended)
```
BRONZE (Raw) → Exact source copy, schema-on-read, partitioned by ingestion
↓ Cleaning, Deduplication
SILVER (Cleansed) → Validated, standardized, business logic applied
↓ Aggregation, Enrichment
GOLD (Business) → Dimensional models, aggregates, ready for BI/ML
```
### Lambda vs Kappa
- **Lambda**: Batch + Stream layers → merged serving layer (complex but complete)
- **Kappa**: Stream-only with replay → simpler but requires robust streaming
## Reference Examples
Full implementation examples in `./references/`:
| File | Description |
|------|-------------|
| `dbt-project-structure.md` | Complete dbt layout with staging, intermediate, marts |
| `airflow-dag.py` | Production DAG with sensors, task groups, quality checks |
| `spark-streaming.py` | Kafka-to-Delta processor with windowing |
| `great-expectations-suite.json` | Comprehensive data quality expectation suite |
## Anti-Patterns (10 Critical Mistakes)
### 1. Full Table Refreshes
**Symptom**: Truncate and rebuild entire tables every run
**Fix**: Use incremental models with `is_incremental()`, partition by date
### 2. Tight Coupling to Source Schemas
**Symptom**: Pipeline breaks when upstream adds/removes columns
**Fix**: Explicit source contracts, select only needed columns in staging
### 3. Monolithic DAGs
**Symptom**: One 200-task DAG running 8 hours
**Fix**: Domain-specific DAGs, ExternalTaskSensor for dependencies
### 4. No Data Quality Gates
**Symptom**: Bad data reaches production before detection
**Fix**: Great Expectations or dbt tests at each layer, block on failures
### 5. Processing Before Archiving
**Symptom**: Raw data transformed without preserving original
**Fix**: Always land raw in Bronze first, make transformations reproducible
### 6. Hardcoded Dates in Queries
**Symptom**: Manual updates needed for date filters
**Fix**: Use Airflow templating (e.g., `ds` variable) or dynamic date functions
### 7. Missing Watermarks in Streaming
**Symptom**: Unbounded state growth, OOM in long-running jobs
**Fix**: Add `withWatermark()` to handle late-arriving data
### 8. No Retry/Backoff Strategy
**Symptom**: Transient failures cause DAG failures
**Fix**: `retries=3`, `retry_exponential_backoff=True`, `max_retry_delay`
### 9. Undocumented Data Lineage
**Symptom**: No one knows where data comes from or who uses it
**Fix**: dbt docs, data catalog integration, column-level lineage
### 10. Testing Only in Production
**Symptom**: Bugs discovered by stakeholders, not engineers
**Fix**: dbt `--target dev`, sample datasets, CI/CD for models
## Quality Checklist
**Pipeline Design:**
- [ ] Incremental processing where possible
- [ ] Idempotent transformations (re-runnable safely)
- [ ] Partitioning strategy defined and documented
- [ ] Backfill procedures documented
**Data Quality:**
- [ ] Tests at Bronze layer (schema, nulls, ranges)
- [ ] Tests at Silver layer (business rules, referential integrity)
- [ ] Tests at Gold layer (aggregation checks, trend monitoring)
- [ ] Anomaly detection for volumes and distributions
**Orchestration:**
- [ ] Retry and alerting configured
- [ ] SLAs defined and monitored
- [ ] Cross-DAG dependencies use sensors
- [ ] max_active_runs prevents parallel conflicts
**Operations:**
- [ ] Data lineage documented
- [ ] Runbooks for common failures
- [ ] Monitoring dashboards for pipeline health
- [ ] On-call procedures defined
## Validation Script
Run `./scripts/validate-pipeline.sh` to check:
- dbt project structure and conventions
- Airflow DAG best practices
- Spark job configurations
- Data quality setup
## External Resources
- [dbt Best Practices](https://docs.getdbt.com/guides/best-practices)
- [Airflow Best Practices](https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html)
- [Great Expectations Docs](https://docs.greatexpectations.io/)
- [Delta Lake Guide](https://docs.delta.io/latest/index.html)
- [Kafka Streams](https://kafka.apache.org/documentation/streams/)
© curiositech · MIT · curiositech/some_claude_skills