Skip to main content
subagent CX Codex

data_engineer — Claude Code subagent for data pipelines

The data_engineer Claude Code subagent designs batch and streaming data pipelines from stated requirements. It returns a Markdown specification covering data flow, quality checks, and operations.

no license ★ 776 updated 2026-11-12

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/agents/data_engineer.md
Open on GitHub

What data_engineer does

The subagent turns requirements for sources, consumers, transformations, freshness, scale, quality, and governance into a pipeline design. Optional context can include samples, schemas, existing jobs, storage details, metrics, or evidence of failures.

Its Markdown specification maps source-to-target flow and contracts, then describes transformations, validation, deduplication, partitioning, and backfills. It also covers scheduling or streaming behavior, retries, checkpoints, observability, lineage, privacy, rollout, and cost.

Schema changes, idempotency, replay, and late-arriving data need explicit treatment. The subagent does not claim data quality or throughput without supplied measurements. Suggested dependencies are routing hints, not a fixed requirement list.

What it helps with

  • map data flow from source to consumer
  • specify transformations and data-quality checks
  • define retry and late-data behavior
  • plan a data backfill

Example prompts for the agent

  • Design a batch pipeline from these requirements: ...
  • Draft a streaming pipeline spec using the schemas and metrics in this project
  • Write a pipeline specification that covers quality checks, retries, and late-arriving data

How to install data_engineer

Manually: put the files in place

.claude/agents/data_engineer.md

For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about data_engineer

How do I install data_engineer in Claude Code?

Add the agent file at .claude/agents/data_engineer.md.

What does data_engineer do?

It produces a Markdown design for a batch or streaming data pipeline, including flow, transformations, quality checks, observability, and backfills.

What requirements does data_engineer need to design a pipeline?

Provide requirements for sources, consumers, transformations, freshness, scale, quality, and governance. Samples, schemas, and details of existing jobs can be added as context.

Similar subagents

All subagents for Data and databases →

From the same set24

The whole set →
web_navigation The web_navigation Claude Code skill reads public web pages and gathers evidence for specific facts. It follows relevant links to find supporting details. CX Codex +1 ★ 776repo stars frontend_developer The frontend_developer Claude Code subagent turns UI requirements and repository context into an implementation plan and code-ready changes. It covers responsive layouts, accessibility, and interface states. CX Codex ★ 776repo stars security_auditor security_auditor — a subagent for Claude Code runs defensive application security reviews. Inspects source code, configurations, and dependency scanner logs to report exploitable flaws. CX Codex ★ 776repo stars search_specialist The search_specialist Claude Code subagent turns an information need into focused queries and assesses retrieved sources. It returns a compact evidence set in JSON for downstream work. CX Codex ★ 776repo stars product_manager product_manager is a Claude Code subagent that turns a product question and available evidence into a clear decision. It helps compare options, set priorities, and define how to measure results. CX Codex ★ 776repo stars api_designer The api_designer Claude Code subagent turns product or service requirements into REST or GraphQL API contracts. It runs on API specification tasks. CX Codex ★ 776repo stars database_optimizer database_optimizer — a subagent for Claude Code, Codex, and Cursor reviews database performance evidence and proposes optimizations. It ranks bottlenecks and outlines how to measure proposed changes. CX Codex ★ 776repo stars fullstack_implementation_engineer The fullstack_implementation_engineer Claude Code subagent turns an approved feature spec into a coordinated implementation plan. It covers data, APIs, business logic, UI, migrations, and tests. CX Codex ★ 776repo stars

Project sponsors

Companies that keep this analytics open