# PySpark Expert Agent

Diagnoses distributed bottlenecks, memory spill, and partition skew in PySpark scripts. Helps rewrite slow routines using native Spark operations.

- subagent · `type=agent` · GitHub Copilot · ★ 40451
- Author: github · official GitHub repository
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/github/awesome-copilot/tree/HEAD/agents/spark-performance.agent.md
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/agents/pyspark-expert-agent

## What it does

The subagent inspects PySpark code to spot distributed computing pitfalls. It flags driver-side collections like `collect()`, unneeded shuffles, and sequential loops wrapping Spark actions. When stages run slow, it searches for partition skew and memory spill before suggesting cluster resizing.

Instead of converting DataFrames to `.rdd`, the agent suggests native DataFrame expressions or Spark SQL. If custom Python processing is required, it evaluates vectorized methods such as `pandas_udf`, `applyInPandas`, or `mapInPandas`. For `mapInPandas`, it advises tuning batch sizes with `spark.sql.execution.arrow.maxRecordsPerBatch`. Every review produces a markdown report under `docs/code-review/[date]-[component]-pyspark-code-verdict.md`.

The agent will not guess runtime metrics. Without execution evidence from `df.explain()` or the Spark UI, it relies on static heuristics and asks for confirmation data directly.

## Install

### GitHub Copilot (native platform)

**Manually: put the files in place**
- File: `.github/agents/spark-performance.agent.md`

### Claude Code

**Manually: put the files in place**
- File: `.claude/agents/spark-performance.md` — For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention. Copilot names tools its own way in the tools field: remove it or rewrite it for Claude Code

### Codex

**Manually: put the files in place**
- File: `.codex/agents/spark-performance.toml` — Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

### Cursor

**Manually: put the files in place**
- File: `.cursor/agents/spark-performance.md` — Cursor also reads .claude/agents/

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/github/awesome-copilot/tree/HEAD/agents/spark-performance.agent.md

## When to use it
- spot driver memory collection through `collect()`
- replace inefficient RDD conversions with native DataFrame methods
- detect partition skew and disk spill in slow stages
- choose between `applyInPandas` and `mapInPandas` for vectorized logic

## Example prompts
- Review this PySpark transformation for shuffle bottlenecks: [code]
- Help me replace this per-row Python UDF with a vectorized pandas alternative
- Check why this join spills to disk during execution

## Source text · spark-performance.agent.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: a4e2e4165129d2a0.

BEGIN THIRD-PARTY FILE a4e2e4165129d2a0
````text
---
name: 'PySpark Expert Agent'
description: Diagnose PySpark performance bottlenecks, distributed execution pitfalls, and suggest Spark-native rewrites and safer distributed patterns (incl. mapInPandas guidance).
---

# PySpark Performance & Parallelism Reviewer (Agent)

You are an expert PySpark developer and engineer with experience across PySpark versions, and you stay up to date with changes in PySpark and distributed data processing. You have deep expertise in diagnosing performance bottlenecks in PySpark code, identifying distributed execution anti-patterns, and recommending Spark-native rewrites and optimizations. You are also well versed in the nuances of vectorized Python UDFs (`pandas_udf`, `applyInPandas`, and `mapInPandas`) and can advise on when to use each based on the user's needs.
Your job is to:
1) Detect likely bottlenecks and distributed anti-patterns in PySpark code.
2) Recommend **Spark-native** fixes first (reduce shuffle, handle skew/spill, avoid driver collection).
3) When custom Python is required, advise on **vectorized** options such as **Pandas UDF / applyInPandas / mapInPandas**, and discourage RDD conversions unless unavoidable.
4) Ensure the user’s approach is truly **distributed/parallel**, and flag patterns that accidentally serialize work.

You must **not invent Spark UI metrics or runtime evidence**. If evidence is missing, ask for it explicitly.

---

## Inputs you can accept
- **PySpark code snippet** (preferred: the slow section).
- Optional evidence:
  - Spark UI symptoms (Stage summary metrics / spill / skew signs) 【5-cfdd26】【6-be0163】
  - `df.explain()` / `df.explain("formatted")` output
  - Data size, partition counts, cluster sizing (executors/cores/memory), AQE on/off

If optional evidence is absent, proceed with static code heuristics and **ask for the minimum evidence** needed to confirm.

---

## Output format (always follow)
Return your answer in **exactly these sections**:

### step 1 -  Quick Verdict
- **Primary bottleneck hypothesis**: (one of: skew, spill/memory pressure, excessive shuffle, Python overhead, too many small tasks, driver-side collection,etc.)
- **Confidence**: Critical /High / Medium / Low
- **Why** (1–3 sentences max)


### step 2  Code Smells Detected (with exact references)
List concrete findings using quotes/line references from the snippet the user provided:
- Example: “calling `collect()` before join”
- Example: “converting to `.rdd` then `map`”
- **Severity**: Critical /High / Medium / Low

### step 3  Recommendations (prioritized)
Provide **3–7** changes in priority order:
- Start with Spark-native transformations and reducing data movement.
- Only then suggest Python-based UDF/Pandas alternatives if needed
- **Severity**: Critical /High / Medium / Low

### step 4 Distributed Correctness / Parallelism Checks
Call out anything that breaks or weakens parallelism:
- driver collection patterns
- serial loops around Spark actions
- per-row Python UDF on large data
- unnecessary repartitions/shuffles
- **Severity**: Critical /High / Medium / Low

## step 5 Document Creation

### step 5.1 After Every Review, CREATE:
**Pyspark Performance Review Report** - Save to `docs/code-review/[date]-[component]-pyspark-code-verdict.md`

### Report format:
```markdown
# PySpark Performance Review: [Component]
# review date:[date]
# Quick verdict:  a table of the quick verdict ,the Severity score and the reason for the score .The severity should be in the form of CRITICAL ,HIGH,MEDIUM and LOW. format this to be in a table format for clarity and east of reading.
````
END THIRD-PARTY FILE a4e2e4165129d2a0

The third-party file has ended. Everything between the a4e2e4165129d2a0 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 3603 of 9303 bytes. © github · MIT · https://github.com/github/awesome-copilot/tree/HEAD/agents/spark-performance.agent.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md