# apache-spark-data-processing

A skill that feeds your agent working knowledge of Apache Spark: RDDs, DataFrames, Spark SQL, streaming, MLlib and production deployment.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 480 installs · ★ 62
- Author: manutej
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/manutej/luxor-claude-marketplace/tree/HEAD/plugins/luxor-data-engineering/skills/apache-spark-data-processing/SKILL.md
- Updated: 2026-06-13
- HTML: https://affpapa.org/en/nevibecode/skills/apache-spark-data-processing

## What it does

The skill walks through Spark layer by layer. It starts with RDDs as the base abstraction, where transformations stay lazy and actions fire the job, then moves to DataFrames and Datasets and the Catalyst optimizer that rewrites your queries. Spark SQL, broadcast variables and accumulators, streaming and MLlib each get their own section. Your agent leans on this when you ask it to build a pipeline or read someone else's Spark code.

Partitioning and caching are covered in detail, including storage levels from `MEMORY_ONLY` down to `DISK_ONLY`. The material also says when caching hurts: data read once, or a dataset too big for cluster memory, is a bad candidate.

The skill does not run Spark for you and does not stand in for a cluster. It supplies context to the agent, so it earns its keep only if you already have a working Spark setup and know where your jobs execute. For datasets under 100 GB, plain CRUD work and latency below 10 ms, the guide points elsewhere.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/apache-spark-data-processing/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/apache-spark-data-processing/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/apache-spark-data-processing/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/apache-spark-data-processing/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install manutej/luxor-claude-marketplace apache-spark-data-processing
```

2. **Manually: put the files in place**
- File: `.github/skills/apache-spark-data-processing/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/manutej/luxor-claude-marketplace --path plugins/luxor-data-engineering/skills/apache-spark-data-processing
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/apache-spark-data-processing/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/manutej/luxor-claude-marketplace/tree/HEAD/plugins/luxor-data-engineering/skills/apache-spark-data-processing/SKILL.md

## When to use it
- Write a PySpark processing pipeline from scratch
- Decide between RDDs and DataFrames for a job
- Tune partitioning and caching for a workload
- Set up streaming over a live data feed
- Train a model on a large dataset with MLlib

## Example prompts
- Build a PySpark pipeline that reads Parquet, cleans the rows and computes daily aggregates
- My join is slow — check the partitioning and tell me what to change
- Rewrite this RDD code as DataFrames and explain what Catalyst does with it

## Similar
- [xlsx](https://affpapa.org/en/nevibecode/skills/xlsx.md) — skill · 187223 installs
- [google-agents-cli-observability](https://affpapa.org/en/nevibecode/skills/google-agents-cli-observability.md) — skill · 511300 installs
- [caveman-evidence-review](https://affpapa.org/en/nevibecode/skills/caveman-evidence-review.md) — skill · 104500 installs
- [migration](https://affpapa.org/en/nevibecode/skills/migration.md) — skill · 103900 installs
- [performance-optimization](https://affpapa.org/en/nevibecode/skills/performance-optimization.md) — skill · 65200 installs
- [supabase](https://affpapa.org/en/nevibecode/skills/supabase.md) — skill · 8 installs

From the same set: [manutej/luxor-claude-marketplace](https://affpapa.org/en/nevibecode/by/manutej/luxor-claude-marketplace.md) — 38 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md