Skip to main content
subagent CX Codex

data-engineer · qdhenry — Claude Code subagent: setup and tasks

The data-engineer subagent designs data pipelines, warehouses, and streaming systems. Use it for ETL/ELT work, Spark jobs, Airflow flows, or Kafka architectures.

no license ★ 1,357 updated 2026-03-01

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/agents/data-engineer.md
Open on GitHub

What data-engineer does

data-engineer is aimed at data-heavy engineering work in Claude Code. It can shape an ETL or ELT pipeline, plan a warehouse, investigate a failing pipeline, and suggest changes to existing infrastructure. Its stated focus includes Apache Spark, Apache Airflow, Apache Kafka, data modeling, distributed processing, and data quality.

The subagent can inspect and edit project files with Read, Write, Edit, MultiEdit, Grep, Glob, Bash, and LS. WebSearch, WebFetch, and the context7 MCP tools support documentation research; sequential-thinking is available for involved design and troubleshooting tasks. Its working rules call for tests, explicit error handling, documentation, and attention to infrastructure cost.

Projects that use none of Spark, Airflow, or Kafka will leave part of this subagent’s specialization unused.

What it helps with

  • design an ETL or ELT pipeline
  • trace a data pipeline failure
  • model a data warehouse
  • plan Kafka-based streaming
  • tune a Spark data job

Example prompts for the agent

  • Design an ETL pipeline for this project and map the task dependencies.
  • Investigate the current pipeline failure and propose a code-level fix.
  • Create a warehouse schema for these data sources and explain the model.

How to install data-engineer

Manually: put the files in place

.claude/agents/data-engineer.md

For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about data-engineer

How do I install data-engineer in Claude Code?

Place the file at .claude/agents/data-engineer.md.

What does data-engineer do?

It helps design and improve ETL/ELT pipelines, warehouses, and streaming systems. It also covers data modeling, quality checks, and troubleshooting for data infrastructure.

Which technologies does data-engineer cover?

Its focus includes Apache Spark, Apache Airflow, and Apache Kafka. Documentation research and structured problem analysis can use context7 and sequential-thinking.

Similar subagents

All subagents for Data and databases →

data-engineer by other authors comes first.

data-engineer VoltAgent Community data-engineer designs data pipelines, ETL/ELT workflows, and data infrastructure. Use it when a project needs pipeline orchestration, data quality checks, or processing cost analysis. CX Codex 3,194installs data-engineer Seth Hobson data-engineer — a subagent for data pipelines, warehouses, and streaming architectures. Use it for Apache Spark, dbt, Airflow, and cloud data platform work. CX Codex 3,194installs data-engineer VoltAgent The data-engineer subagent investigates ETL, ingestion, and pipeline failures, then proposes a scoped fix. Use it when data contracts or reprocessing behavior need a closer look. CX Codex ★ 6,505repo stars data-engineer josstei data-engineer designs database schemas, optimizes SQL queries, and builds ETL pipelines. Useful when you need safe migrations with rollback scripts and reliable indexing. CX Codex ★ 489repo stars senior-data-engineer Alireza Rezvani senior-data-engineer is a Claude Code skill for designing data pipelines and ETL/ELT systems. It activates for architecture choices, orchestration, data quality checks, and troubleshooting. CX Codex +1 1,109installs data-pipeline-engineer curiositech data-pipeline-engineer guides data pipeline design, Medallion storage layers, Airflow DAG orchestration, and dbt models. Activates for batch processing and streaming workloads. CX Codex +1 215installs data-engineering travisjneuman The data-engineering Claude Code skill covers data pipelines and warehouse schema design. It activates for ETL/ELT work, stream processing, and dbt transformations. CX Codex +1 105installs discover-data rand discover-data is a Claude Code skill that loads data pipeline and ETL guides. It triggers on tasks with Airflow, stream processing, batch pipelines, and data validation. CX Codex +1 101installs data-pipeline Seth Hobson data-pipeline is a Claude Code command that designs and builds batch and streaming data pipelines. It covers ingestion, transformations, quality checks, and monitoring. CX Codex +1 ★ 42,780repo stars data-engineering-study-material reason-machines data-engineering-study-material — a study guide skill for data engineering: explains concepts, shows pipeline examples, and helps with interview prep. CX Codex +1 2,000installs

From the same set112

The whole set →
semantic-architect semantic-architect is a Claude Code subagent that organizes dialogue context into tree structures and preserves persistent memory across sessions. CX Codex ★ 1,357repo stars golang-pro golang-pro — a Go-focused subagent for architecture, implementation, refactoring, concurrency, APIs, testing, and performance work in Claude Code. CX Codex ★ 1,357repo stars architect-reviewer architect-reviewer — a Claude subagent that reads a change for architectural fit: patterns, SOLID, dependency direction. Useful after structural edits or a new service. CX Codex ★ 1,357repo stars orchestration-readme orchestration-readme breaks large features into structured task files, tracks active stages, and aligns code changes with Git commits inside Claude Code. CX Codex +1 ★ 1,357repo stars database-optimizer database-optimizer — a subagent for database performance analysis: finds bottlenecks in SQL queries, indexing, schema design and infrastructure. CX Codex ★ 1,357repo stars ui-designer ui-designer — a Claude Code subagent for building interface mockups, design systems, and accessible layouts using magic and context7 MCP servers. CX Codex ★ 1,357repo stars cloudflare-worker Generate a Cloudflare Worker from a task description, with Wrangler configuration and sample requests. Useful for building an API, WebSocket service, or authentication middleware. CX Codex +1 ★ 1,357repo stars legacy-modernizer legacy-modernizer is a Claude Code subagent for planning and carrying out gradual changes to legacy systems. It engages on framework upgrades, codebase refactors, and monolith decomposition. CX Codex ★ 1,357repo stars

Project sponsors

Companies that keep this analytics open