# data-engineer

This skill helps plan data pipelines, warehouses, and streaming systems. Use it when a project needs ingestion, orchestration, or data quality checks.

- skill · `type=skill` · Claude Code · 152 installs
- Author: davila7
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/data-engineer
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/data-engineer

## What it does

The skill gives Claude Code a data-engineering workflow. It starts with sources, SLAs, and data contracts, then covers architecture, storage, orchestration, ingestion, transformation, validation, and monitoring. The guidance spans batch and streaming designs.

Its listed stack includes Apache Spark, dbt, Apache Airflow, Kafka, Flink, lakehouse technologies, and data services from AWS, Azure, and GCP. It also covers warehouse modeling, lineage, governance, access control, and checks before data reaches production sinks.

The project needs access to data sources or storage systems. That limitation matters: the skill is not intended for exploratory analysis alone or for ML model work without pipelines.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/data-engineer
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill data-engineer -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/data-engineer/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/data-engineer/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/data-engineer/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/data-engineer/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates data-engineer
```

2. **Manually: put the files in place**
- File: `.github/skills/data-engineer/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/data-engineer
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/data-engineer/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/data-engineer

## When to use it
- design a batch or streaming data pipeline
- plan a warehouse or lakehouse architecture
- map ingestion, transformation, and validation stages
- choose orchestration with Apache Airflow
- set up data quality and lineage checks

## Example prompts
- Design a batch pipeline and define its sources, SLAs, data contracts, and validation stages.
- Compare lakehouse options built around Apache Spark, dbt, and Apache Airflow, including data quality checks.
- Plan Kafka and Flink stream processing with PII protection and access control.

## Similar
- [xlsx](https://affpapa.org/en/nevibecode/skills/xlsx.md) — skill · 187223 installs
- [google-agents-cli-observability](https://affpapa.org/en/nevibecode/skills/google-agents-cli-observability.md) — skill · 511300 installs
- [caveman-evidence-review](https://affpapa.org/en/nevibecode/skills/caveman-evidence-review.md) — skill · 104500 installs
- [migration](https://affpapa.org/en/nevibecode/skills/migration.md) — skill · 103900 installs
- [performance-optimization](https://affpapa.org/en/nevibecode/skills/performance-optimization.md) — skill · 65200 installs
- [supabase](https://affpapa.org/en/nevibecode/skills/supabase.md) — skill · 8 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 495 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md