# data-engineer

The data-engineer subagent designs data pipelines, warehouses, and streaming systems. Use it for ETL/ELT work, Spark jobs, Airflow flows, or Kafka architectures.

- subagent · `type=agent` · Claude Code, Codex, Cursor · ★ 1357 repo stars
- Author: qdhenry
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/qdhenry/Claude-Command-Suite/tree/HEAD/.claude/agents/data-ai/data-engineer.md
- Updated: 2026-03-01
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/agents/qdhenry-data-engineer

## What it does

data-engineer is aimed at data-heavy engineering work in Claude Code. It can shape an ETL or ELT pipeline, plan a warehouse, investigate a failing pipeline, and suggest changes to existing infrastructure. Its stated focus includes Apache Spark, Apache Airflow, Apache Kafka, data modeling, distributed processing, and data quality.

The subagent can inspect and edit project files with Read, Write, Edit, MultiEdit, Grep, Glob, Bash, and LS. WebSearch, WebFetch, and the `context7` MCP tools support documentation research; `sequential-thinking` is available for involved design and troubleshooting tasks. Its working rules call for tests, explicit error handling, documentation, and attention to infrastructure cost.

Projects that use none of Spark, Airflow, or Kafka will leave part of this subagent’s specialization unused.

## Install

### Claude Code (native platform)

**Manually: put the files in place**
- File: `.claude/agents/data-engineer.md` — For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

### Codex

**Manually: put the files in place**
- File: `.codex/agents/data-engineer.toml` — Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

### Cursor

**Manually: put the files in place**
- File: `.cursor/agents/data-engineer.md` — Cursor also reads .claude/agents/

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/qdhenry/Claude-Command-Suite/tree/HEAD/.claude/agents/data-ai/data-engineer.md

## When to use it
- design an ETL or ELT pipeline
- trace a data pipeline failure
- model a data warehouse
- plan Kafka-based streaming
- tune a Spark data job

## Example prompts
- Design an ETL pipeline for this project and map the task dependencies.
- Investigate the current pipeline failure and propose a code-level fix.
- Create a warehouse schema for these data sources and explain the model.

## Similar
- [senior-data-engineer](https://affpapa.org/en/nevibecode/skills/senior-data-engineer.md) — skill · 1109 installs
- [data-pipeline-engineer](https://affpapa.org/en/nevibecode/skills/data-pipeline-engineer.md) — skill · 215 installs
- [data-engineering](https://affpapa.org/en/nevibecode/skills/travisjneuman-data-engineering.md) — skill · 105 installs
- [discover-data](https://affpapa.org/en/nevibecode/skills/discover-data.md) — skill · 101 installs
- [data-pipeline](https://affpapa.org/en/nevibecode/commands/data-pipeline.md) — command · ★ 42780 repo stars
- [data-engineering-study-material](https://affpapa.org/en/nevibecode/skills/data-engineering-study-material.md) — skill · 2000 installs

## Other versions with the same name
- [data-engineer · voltagent](https://affpapa.org/en/nevibecode/agents/voltagent-community-data-engineer.md) — 3194 installs
- [data-engineer · wshobson](https://affpapa.org/en/nevibecode/agents/data-engineer.md) — 3194 installs
- [data-engineer · voltagent](https://affpapa.org/en/nevibecode/agents/voltagent-data-engineer.md) — ★ 6505 repo stars
- [data-engineer · josstei](https://affpapa.org/en/nevibecode/agents/josstei-data-engineer.md) — ★ 489 repo stars

From the same set: [qdhenry/claude-command-suite](https://affpapa.org/en/nevibecode/by/qdhenry/claude-command-suite.md) — 112 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
