# executing-spark

executing-spark is a Claude Code skill that runs Python or PySpark on Fabric Spark through Livy without creating a notebook. It supports temporary transformations and lakehouse table queries.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf (Devin Desktop), Claude.ai · ★ 1104 repo stars
- Author: Kurt Buhler
- License: GPL-3.0 · policy=link-only · description only; the file stays with the author
- Source: https://github.com/data-goblin/power-bi-agentic-development/tree/HEAD/plugins/etl/skills/executing-spark/SKILL.md
- Updated: 2026-10-28
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/executing-spark

## What it does

The skill opens a Livy session, sends code to it, and returns the execution result. The session can use `spark.sql`, Delta tables, and regular Python; multiple statements can run before it closes. No notebook is saved.

You need Azure CLI authentication with `az login`, a lakehouse in the target workspace, and Fabric capacity on F or trial. Get the token with `az account get-access-token --resource https://api.fabric.microsoft.com`; a `fab auth` token cannot access OneLake from the Spark session. Close the session when the work is done, since an idle one still uses Fabric capacity units.

## Requirements
- Works with: Power BI

## Install

### Claude Code (native platform)

1. **Plugin marketplace**
In the agent chat (not a shell):
```text
/plugin marketplace add data-goblin/power-bi-agentic-development
```
In the agent chat (not a shell):
```text
/plugin install etl@power-bi-agentic-development
```
This installs the whole etl plugin with all its skills

2. **Skills CLI (Vercel)**
```bash
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/executing-spark/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/executing-spark/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/executing-spark/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/executing-spark/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install data-goblin/power-bi-agentic-development plugins/etl/skills/executing-spark
```

2. **Manually: put the files in place**
- File: `.github/skills/executing-spark/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/data-goblin/power-bi-agentic-development --path plugins/etl/skills/executing-spark
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/executing-spark/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/data-goblin/power-bi-agentic-development/tree/HEAD/plugins/etl/skills/executing-spark/SKILL.md

## When to use it
- run PySpark on Fabric without creating a notebook
- query lakehouse tables with Spark SQL
- write transformed data to a Delta table
- execute several statements in one Livy session

## Example prompts
- Run this PySpark code on Fabric through Livy without creating a notebook.
- Query the lakehouse tables with `spark.sql` and show the result.
- Transform the data in Spark, write it to a Delta table, then close the session.

## Similar
- [sql-jdbc-access-design](https://affpapa.org/en/nevibecode/skills/sql-jdbc-access-design.md) — skill · 10100 installs
- [powerbi-modeling](https://affpapa.org/en/nevibecode/skills/powerbi-modeling.md) — skill · 9934 installs
- [power-bi-report-design-consultation](https://affpapa.org/en/nevibecode/skills/power-bi-report-design-consultation.md) — skill · 9390 installs
- [power-bi-build](https://affpapa.org/en/nevibecode/skills/power-bi-build.md) — skill · 688 installs
- [power-bi-salespulse-360-dashboard](https://affpapa.org/en/nevibecode/skills/power-bi-salespulse-360-dashboard.md) — skill · 661 installs
- [logifleet-pulse-supply-chain-analytics](https://affpapa.org/en/nevibecode/skills/logifleet-pulse-supply-chain-analytics.md) — skill · 659 installs

From the same set: [data-goblin/power-bi-agentic-development](https://affpapa.org/en/nevibecode/by/data-goblin/power-bi-agentic-development.md) — 59 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
