Skip to main content
affpapa.org
RU EN
skill CX Codex

apache-spark-data-processing — Claude Code skill for Apache Spark

A skill that feeds your agent working knowledge of Apache Spark: RDDs, DataFrames, Spark SQL, streaming, MLlib and production deployment.

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓ 
npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a claude-code
Open on GitHub

What apache-spark-data-processing does

The skill walks through Spark layer by layer. It starts with RDDs as the base abstraction, where transformations stay lazy and actions fire the job, then moves to DataFrames and Datasets and the Catalyst optimizer that rewrites your queries. Spark SQL, broadcast variables and accumulators, streaming and MLlib each get their own section. Your agent leans on this when you ask it to build a pipeline or read someone else's Spark code.

Partitioning and caching are covered in detail, including storage levels from MEMORY_ONLY down to DISK_ONLY. The material also says when caching hurts: data read once, or a dataset too big for cluster memory, is a bad candidate.

The skill does not run Spark for you and does not stand in for a cluster. It supplies context to the agent, so it earns its keep only if you already have a working Spark setup and know where your jobs execute. For datasets under 100 GB, plain CRUD work and latency below 10 ms, the guide points elsewhere.

What it helps with

  • Write a PySpark processing pipeline from scratch
  • Decide between RDDs and DataFrames for a job
  • Tune partitioning and caching for a workload
  • Set up streaming over a live data feed
  • Train a model on a large dataset with MLlib

Example prompts for the agent

  • Build a PySpark pipeline that reads Parquet, cleans the rows and computes daily aggregates
  • My join is slow — check the partitioning and tell me what to change
  • Rewrite this RDD code as DataFrames and explain what Catalyst does with it

How to install apache-spark-data-processing

1Skills CLI (Vercel)

npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a claude-code

2Manually: put the files in place

~/.claude/skills/apache-spark-data-processing/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/apache-spark-data-processing/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about apache-spark-data-processing

How do I install apache-spark-data-processing in Claude Code?

Run npx skills add manutej/luxor-claude-marketplace --skill apache-spark-data-processing -a claude-code. The skill file then sits at ~/.claude/skills/apache-spark-data-processing/SKILL.md.

What does apache-spark-data-processing do?

It gives the agent a reference on Apache Spark: RDDs, DataFrames, Spark SQL, streaming, MLlib and deployment. The agent pulls from it while you write or review data processing code.

When should you avoid Spark?

For datasets under 100 GB, simple CRUD operations and workloads needing latency below 10 ms. The guide recommends other tools in those cases.

Similar skills

All skills for Data and databases →
xlsx official anthropics Creates, reads, edits, and converts spreadsheet files. Use it when the requested deliverable is an Excel workbook or another tabular file. 187,223installs google-agents-cli-observability official Google This skill guides you through tracing and monitoring AI agents, including ADK agents. Use it when you need to inspect agent execution or monitor a deployed agent. CX Codex +1 511,300installs caveman-evidence-review JuliusBrussee A read-only skill reviews Caveman Cloud evidence on LLM spend, traces, and routing. Use it to see where a project’s model budget goes. CX Codex +1 104,500installs migration JuliusBrussee Enforces reversible, compatibility-safe transitions for schema, data, and API updates. Useful whenever data loss or rollout downtime is unacceptable. CX Codex +1 103,900installs performance-optimization addyosmani An agent skill that measures before it changes code: Core Web Vitals budgets, N+1 queries, missing indexes and slow endpoints. CX Codex +1 65,200installs supabase official supabase Guides the agent through database configuration and strict RLS policies in Supabase. CX Codex GP ChatGPT 8installs

From the same set38

rest-api-design-patterns A reference skill for designing REST APIs: resource modeling, versioning, pagination, HATEOAS and HTTP method semantics. CX Codex +1 525installs asyncio-concurrency-patterns A skill on Python concurrency: event loops, coroutines, tasks, semaphores and locks. Useful when you are writing an async service or client. CX Codex +1 489installs rust-systems-programming A skill on systems Rust: ownership, borrowing, threads, async, unsafe and performance. Useful when you write low-level code or port C/C++. CX Codex +1 485installs api-gateway-patterns A skill for building API gateways on Kong: routing, rate limiting, auth, load balancing. Useful when your microservices exist but lack a single entry point. CX Codex +1 481installs nodejs-development A Node.js reference skill for your coding agent: event loop phases, promises, streams, file system. Useful when the agent writes backend code or CLI tools. CX Codex +1 472installs react-patterns A skill that hands your agent ready React recipes: hooks, component composition, state management and fewer wasted re-renders. CX Codex +1 471installs SQLAlchemy ORM Expert A SQLAlchemy 2.0+ skill for customer support backends: ticket models, sessions, async queries and PostgreSQL. Useful when you build on FastAPI. CX Codex +1 470installs pandas A skill that brings pandas expertise to support operations: SLA compliance, agent metrics, PostgreSQL pulls. Useful when you need to dig through tickets and build a report. CX Codex +1 469installs

Project sponsors

Companies that keep this analytics open
Gold sponsor
Nutra.Club

Impressions, clicks and code copies — open statistics. We keep the numbers public for the same reason NeBlask itself is public. Become a sponsor