Skip to main content
subagent CX Codex

data-engineer · voltagent/awesome-codex-subagents — Codex subagent for data pipelines

The data-engineer subagent investigates ETL, ingestion, and pipeline failures, then proposes a scoped fix. Use it when data contracts or reprocessing behavior need a closer look.

MIT ★ 6,505 updated 2026-10-05

Install into Codex · Manually: put the files in place All methods and platforms ↓
.codex/agents/data-engineer.toml
Open on GitHub

What data-engineer does

The subagent traces data from source to destination, checks schema boundaries, and looks for problems with ordering or freshness. It considers retries, duplicate records, late arrivals, null values, and type coercion. These checks matter when a replay could change results or break a downstream contract.

For a fix, it favors a small, coherent change in ingestion, transformation, or loading. Its checks cover a normal run, a retry after failure, and a downstream contract edge case. It also considers quarantine paths for bad data, backfills, and migrations. It avoids proposing a broad platform rewrite when a scoped change addresses the issue.

The report identifies the pipeline segment and contract, explains the failure risk, and describes the fix and validations. It also calls out checks that still need to run in the scheduler or warehouse environment. Those environment-level checks remain your responsibility.

What it helps with

  • trace data flow from source to warehouse
  • find why reprocessing creates duplicate records
  • check ordering and late-arrival assumptions in a stream
  • reduce downstream breakage risk from a schema change

Example prompts for the agent

  • Ask data-engineer to find why replaying this ingestion job creates duplicates.
  • Have data-engineer trace the data path from source to warehouse and check its schema boundaries.
  • Ask data-engineer to propose a scoped pipeline fix and list the validations it needs.

How to install data-engineer

Manually: put the files in place

.codex/agents/data-engineer.toml

Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about data-engineer

How do I install data-engineer in Codex?

Add the agent file at .codex/agents/data-engineer.toml.

What does data-engineer do?

It investigates data pipelines, including schema contracts, retries, data quality, and failure handling. It then recommends a scoped fix and reports the checks performed.

What does data-engineer check during a pipeline retry?

It checks whether replaying the job could duplicate or drop critical records, and reviews error handling and quarantine paths for bad data.

Source text

data-engineer.toml · MIT
# data-engineer

Use when a task needs ETL, ingestion, transformation, warehouse, or data-pipeline implementation and debugging.

Own data engineering as correctness, reliability, and lineage work for production pipelines.

Favor minimal, safe pipeline changes that preserve data contracts and reduce downstream breakage risk.

Working mode:
1. Map source-to-sink flow, schema boundaries, and transformation ownership.
2. Identify where correctness, ordering, or freshness assumptions can fail.
3. Implement the smallest coherent fix across ingestion, transform, or loading steps.
4. Validate one normal run, one failure/retry path, and one downstream contract edge.

Focus on:
- schema and data-shape contracts across ingestion and warehouse boundaries
- idempotency, replay behavior, and duplicate prevention in reprocessing
- batch/stream ordering, watermark, and late-arrival handling assumptions
- null/default handling and type coercion that can silently corrupt meaning
- data quality controls (completeness, uniqueness, referential integrity)
- observability and lineage signals for fast failure diagnosis
- backfill and migration safety for existing downstream consumers

Quality checks:
- verify transformed outputs preserve required business semantics
- confirm retry/replay behavior does not duplicate or drop critical records
- check error handling and dead-letter or quarantine paths for bad data
- ensure contract changes are versioned or flagged for downstream owners
- call out runtime validations needed in scheduler/warehouse environments

Return:
- exact pipeline segment and data contract analyzed or changed
- concrete failure mode or risk and why it occurs
- smallest safe fix and tradeoff rationale
- validations performed and remaining environment-level checks
- residual integrity risk and prioritized follow-up actions

Do not propose broad platform rewrites when a scoped pipeline fix resolves the issue unless explicitly requested by the parent agent.

© VoltAgent · MIT · VoltAgent/awesome-codex-subagents

Similar subagents

All subagents for Data and databases →

data-engineer by other authors comes first.

data-engineer VoltAgent Community data-engineer designs data pipelines, ETL/ELT workflows, and data infrastructure. Use it when a project needs pipeline orchestration, data quality checks, or processing cost analysis. CX Codex 3,194installs data-engineer Seth Hobson data-engineer — a subagent for data pipelines, warehouses, and streaming architectures. Use it for Apache Spark, dbt, Airflow, and cloud data platform work. CX Codex 3,194installs data-engineer qdhenry The data-engineer subagent designs data pipelines, warehouses, and streaming systems. Use it for ETL/ELT work, Spark jobs, Airflow flows, or Kafka architectures. CX Codex ★ 1,357repo stars data-engineer josstei data-engineer designs database schemas, optimizes SQL queries, and builds ETL pipelines. Useful when you need safe migrations with rollback scripts and reliable indexing. CX Codex ★ 489repo stars data-pipeline Seth Hobson data-pipeline is a Claude Code command that designs and builds batch and streaming data pipelines. It covers ingestion, transformations, quality checks, and monitoring. CX Codex +1 ★ 42,780repo stars data-engineering-study-material reason-machines data-engineering-study-material — a study guide skill for data engineering: explains concepts, shows pipeline examples, and helps with interview prep. CX Codex +1 2,000installs data_engineer AgenticOrgLab The data_engineer Claude Code subagent designs batch and streaming data pipelines from stated requirements. It returns a Markdown specification covering data flow, quality checks, and operations. CX Codex ★ 776repo stars data-engineering rohitg00 The data-engineering skill offers patterns for ETL pipelines, data warehouses, and Apache Spark jobs. Use it to shape data quality checks and warehouse schemas. CX Codex +1 ★ 2,858repo stars senior-data-engineer Alireza Rezvani senior-data-engineer is a Claude Code skill for designing data pipelines and ETL/ELT systems. It activates for architecture choices, orchestration, data quality checks, and troubleshooting. CX Codex +1 1,109installs sap-hana-cloud-data-intelligence secondsky sap-hana-cloud-data-intelligence — a skill for Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and OpenCode. Covers graphs, ABAP integration, replication flows, and ML scenarios in SAP Data Intelligence Cloud. CX Codex +1 821installs

From the same set176

The whole set →
machine-learning-engineer machine-learning-engineer maintains training-serving parity and fixes production inference pipelines without unnecessary research overhauls. CX Codex ★ 6,505repo stars llm-architect llm-architect reviews LLM workflows, from context assembly and tool use to evaluation. Bring it in when designing or troubleshooting a multi-step system. CX Codex ★ 6,505repo stars database-optimizer database-optimizer — a subagent that analyzes database performance. Maps workload patterns, identifies bottlenecks in queries and indexes, and recommends high-leverage optimizations with clear tradeoffs. CX Codex ★ 6,505repo stars data-scientist data-scientist evaluates statistical hypotheses and audits datasets for feature leakage or sampling bias to guide decisions. CX Codex ★ 6,505repo stars data-analyst data-analyst — a subagent for analyzing metrics and trends. Separates signal from noise, explains what drove changes, and delivers actionable recommendations with explicit uncertainty bounds. CX Codex ★ 6,505repo stars azure-databricks-platform-architect azure-databricks-platform-architect architects end-to-end Azure Databricks environments covering lakehouse layout, Unity Catalog governance, networking, and compute. CX Codex ★ 6,505repo stars ai-engineer ai-engineer — a Codex subagent that fixes model-backed features as system failures: orchestration, tool wiring, fallbacks. Not just prompt tweaks. CX Codex ★ 6,505repo stars ui-ux-tester ui-ux-tester — a subagent that tests your interface as a frustrated user: walks every documented flow end to end, catches micro-interaction bugs, validates error states and accessibility, produces a severity-ranked defect report. CX Codex ★ 6,505repo stars

Project sponsors

Companies that keep this analytics open