# senior-data-engineer

senior-data-engineer — a skill for Claude Code and Codex that automates data engineering workflows. Generates orchestrator DAGs, profiles data schemas, and flags query bottlenecks in Spark and SQL.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 199 installs · ★ 1183 repo stars
- Author: borghei
- License: LicenseRef-MIT-Commons-Clause · policy=link-only · description only; the file stays with the author
- Source: https://github.com/borghei/Claude-Skills/tree/HEAD/engineering/senior-data-engineer/SKILL.md
- Updated: 2026-11-08
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/borghei-senior-data-engineer

## What it does

The skill bundles three Python utilities: `pipeline_orchestrator.py`, `data_quality_validator.py`, and `etl_performance_optimizer.py`. Together they assemble batch or incremental DAGs for Airflow, Prefect, and Dagster, while also handling partition design.

Data validation runs against custom JSON contracts or generated Great Expectations suites, checking formats, missing values, and numeric bounds. SQL review via `analyze-sql` relies on static inspection. You still need query execution profiles from your warehouse to confirm actual runtimes and disk spills.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add borghei/Claude-Skills --skill senior-data-engineer -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/senior-data-engineer/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/senior-data-engineer/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add borghei/Claude-Skills --skill senior-data-engineer -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/senior-data-engineer/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add borghei/Claude-Skills --skill senior-data-engineer -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/senior-data-engineer/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install borghei/Claude-Skills engineering/senior-data-engineer/SKILL.md
```

2. **Manually: put the files in place**
- File: `.github/skills/senior-data-engineer/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add borghei/Claude-Skills --skill senior-data-engineer -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/borghei/Claude-Skills --path engineering/senior-data-engineer
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/senior-data-engineer/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add borghei/Claude-Skills --skill senior-data-engineer -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/borghei/Claude-Skills/tree/HEAD/engineering/senior-data-engineer/SKILL.md

## When to use it
- build incremental DAG configurations for Airflow, Prefect, or Dagster
- generate JSON contracts and Great Expectations test suites from tabular data
- detect schema drift and invalid records in CSV datasets
- analyze Spark jobs and warehouse SQL queries to optimize partitioning

## Example prompts
- Generate an incremental Airflow DAG from PostgreSQL to Snowflake for orders and customers.
- Validate `sample_orders_flawed.csv` against `sample_schema.json` with anomaly detection enabled.
- Run static SQL analysis on `assets/sample_query.sql` targeting Snowflake.

## Similar
- [data-pipeline-engineer](https://affpapa.org/en/nevibecode/skills/data-pipeline-engineer.md) — skill · 215 installs
- [data-engineer](https://affpapa.org/en/nevibecode/skills/data-engineer.md) — skill · 152 installs
- [data-engineering](https://affpapa.org/en/nevibecode/skills/travisjneuman-data-engineering.md) — skill · 105 installs
- [oracle-ai-data-platform-workbench-engineer-agent](https://affpapa.org/en/nevibecode/plugins/oracle-ai-data-platform-workbench-engineer-agent.md) — plugin · ★ 62 repo stars
- [realtime-cinema-data-engineering-pipeline](https://affpapa.org/en/nevibecode/skills/realtime-cinema-data-engineering-pipeline.md) — skill · 2000 installs
- [discover-data](https://affpapa.org/en/nevibecode/skills/discover-data.md) — skill · 101 installs

## Other versions with the same name
- [senior-data-engineer · alirezarezvani](https://affpapa.org/en/nevibecode/skills/senior-data-engineer.md) — 1109 installs
- [senior-data-engineer · benchflow-ai](https://affpapa.org/en/nevibecode/skills/claude-skills-team-senior-data-engineer.md) — 1 install

From the same set: [borghei/claude-skills](https://affpapa.org/en/nevibecode/by/borghei/claude-skills.md) — 120 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
