data_engineer — Claude Code subagent for data pipelines
The data_engineer Claude Code subagent designs batch and streaming data pipelines from stated requirements. It returns a Markdown specification covering data flow, quality checks, and operations.
no license ★ 776 updated 2026-11-12
.claude/agents/data_engineer.mdWhat data_engineer does
The subagent turns requirements for sources, consumers, transformations, freshness, scale, quality, and governance into a pipeline design. Optional context can include samples, schemas, existing jobs, storage details, metrics, or evidence of failures.
Its Markdown specification maps source-to-target flow and contracts, then describes transformations, validation, deduplication, partitioning, and backfills. It also covers scheduling or streaming behavior, retries, checkpoints, observability, lineage, privacy, rollout, and cost.
Schema changes, idempotency, replay, and late-arriving data need explicit treatment. The subagent does not claim data quality or throughput without supplied measurements. Suggested dependencies are routing hints, not a fixed requirement list.
What it helps with
- map data flow from source to consumer
- specify transformations and data-quality checks
- define retry and late-data behavior
- plan a data backfill
Example prompts for the agent
- Design a batch pipeline from these requirements:
... - Draft a streaming pipeline spec using the schemas and metrics in this project
- Write a pipeline specification that covers quality checks, retries, and late-arriving data
How to install data_engineer
Manually: put the files in place
.claude/agents/data_engineer.mdFor all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention
Manually: put the files in place
.codex/agents/data_engineer.tomlCodex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/
Manually: put the files in place
.cursor/agents/data_engineer.mdCursor also reads .claude/agents/
Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean
Questions about data_engineer
How do I install data_engineer in Claude Code?
Add the agent file at .claude/agents/data_engineer.md.
What does data_engineer do?
It produces a Markdown design for a batch or streaming data pipeline, including flow, transformations, quality checks, observability, and backfills.
What requirements does data_engineer need to design a pipeline?
Provide requirements for sources, consumers, transformations, freshness, scale, quality, and governance. Samples, schemas, and details of existing jobs can be added as context.