# data_engineer

The data_engineer Claude Code subagent designs batch and streaming data pipelines from stated requirements. It returns a Markdown specification covering data flow, quality checks, and operations.

- subagent · `type=agent` · Claude Code, Codex, Cursor · ★ 776 repo stars
- Author: AgenticOrgLab
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/AgenticOrgLab/ExecOrchestra/tree/HEAD/execorchestra/Pool/resources/agents/data_engineer.md
- Updated: 2026-11-12
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/agents/agenticorglab-data-engineer

## What it does

The subagent turns requirements for sources, consumers, transformations, freshness, scale, quality, and governance into a pipeline design. Optional context can include samples, schemas, existing jobs, storage details, metrics, or evidence of failures.

Its Markdown specification maps source-to-target flow and contracts, then describes transformations, validation, deduplication, partitioning, and backfills. It also covers scheduling or streaming behavior, retries, checkpoints, observability, lineage, privacy, rollout, and cost.

Schema changes, idempotency, replay, and late-arriving data need explicit treatment. The subagent does not claim data quality or throughput without supplied measurements. Suggested dependencies are routing hints, not a fixed requirement list.

## Install

### Claude Code (native platform)

**Manually: put the files in place**
- File: `.claude/agents/data_engineer.md` — For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

### Codex

**Manually: put the files in place**
- File: `.codex/agents/data_engineer.toml` — Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

### Cursor

**Manually: put the files in place**
- File: `.cursor/agents/data_engineer.md` — Cursor also reads .claude/agents/

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/AgenticOrgLab/ExecOrchestra/tree/HEAD/execorchestra/Pool/resources/agents/data_engineer.md

## When to use it
- map data flow from source to consumer
- specify transformations and data-quality checks
- define retry and late-data behavior
- plan a data backfill

## Example prompts
- Design a batch pipeline from these requirements: `...`
- Draft a streaming pipeline spec using the schemas and metrics in this project
- Write a pipeline specification that covers quality checks, retries, and late-arriving data

## Similar
- [data-engineer](https://affpapa.org/en/nevibecode/agents/voltagent-data-engineer.md) — subagent · ★ 6505 repo stars
- [data](https://affpapa.org/en/nevibecode/skills/data.md) — skill · ★ 26 repo stars
- [data-quality-frameworks](https://affpapa.org/en/nevibecode/skills/data-quality-frameworks.md) — skill · 11230 installs
- [data-engineering-study-material](https://affpapa.org/en/nevibecode/skills/data-engineering-study-material.md) — skill · 2000 installs
- [data-engineering](https://affpapa.org/en/nevibecode/skills/data-engineering.md) — skill · ★ 2858 repo stars
- [senior-data-engineer](https://affpapa.org/en/nevibecode/skills/senior-data-engineer.md) — skill · 1109 installs

From the same set: [agenticorglab/execorchestra](https://affpapa.org/en/nevibecode/by/agenticorglab/execorchestra.md) — 24 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
