executing-spark — Claude Code skill for running PySpark in Fabric
executing-spark is a Claude Code skill that runs Python or PySpark on Fabric Spark through Livy without creating a notebook. It supports temporary transformations and lakehouse table queries.
GPL-3.0 ★ 1,104 updated 2026-10-28
/plugin marketplace add data-goblin/power-bi-agentic-development
/plugin install etl@power-bi-agentic-developmentWhat executing-spark does
The skill opens a Livy session, sends code to it, and returns the execution result. The session can use spark.sql, Delta tables, and regular Python; multiple statements can run before it closes. No notebook is saved.
You need Azure CLI authentication with az login, a lakehouse in the target workspace, and Fabric capacity on F or trial. Get the token with az account get-access-token --resource https://api.fabric.microsoft.com; a fab auth token cannot access OneLake from the Spark session. Close the session when the work is done, since an idle one still uses Fabric capacity units.
What it helps with
- run PySpark on Fabric without creating a notebook
- query lakehouse tables with Spark SQL
- write transformed data to a Delta table
- execute several statements in one Livy session
Example prompts for the agent
- Run this PySpark code on Fabric through Livy without creating a notebook.
- Query the lakehouse tables with
spark.sqland show the result. - Transform the data in Spark, write it to a Delta table, then close the session.
How to install executing-spark
1Plugin marketplace
/plugin marketplace add data-goblin/power-bi-agentic-development/plugin install etl@power-bi-agentic-developmentThis installs the whole etl plugin with all its skills
2Skills CLI (Vercel)
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a claude-code3Manually: put the files in place
~/.claude/skills/executing-spark/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/executing-spark/
1Skills CLI (Vercel)
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a codex2Manually: put the files in place
.agents/skills/executing-spark/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a cursor2Manually: put the files in place
.cursor/skills/executing-spark/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install data-goblin/power-bi-agentic-development plugins/etl/skills/executing-spark2Manually: put the files in place
.github/skills/executing-spark/SKILL.md1Skills CLI (Vercel)
npx skills add data-goblin/power-bi-agentic-development --skill executing-spark -a gemini-cli2Platform CLI
gemini skills install https://github.com/data-goblin/power-bi-agentic-development --path plugins/etl/skills/executing-sparkManually: put the files in place
.devin/skills/executing-spark/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about executing-spark
How do I install executing-spark in Claude Code?
Add the marketplace with /plugin marketplace add data-goblin/power-bi-agentic-development, then install the plugin with /plugin install etl@power-bi-agentic-development.
What does executing-spark do?
It sends Python or PySpark to Fabric compute through Livy and returns the result without saving a notebook. The code can read and write lakehouse tables.
What does the skill need to run?
Authenticate Azure CLI with az login, and provide a lakehouse in the target workspace plus Fabric capacity on F or trial. OneLake access needs a token from az account get-access-token; fab auth tokens do not work for that access.