# scala-spark

Instructions for GitHub Copilot covering Scala and Apache Spark development. Helps structure DataFrame pipelines, tune joins, and avoid common memory bottlenecks.

- rule · `type=rule` · GitHub Copilot · ★ 40451
- Author: github · official GitHub repository
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/github/awesome-copilot/tree/HEAD/instructions/scala-spark.instructions.md
- Updated: 2026-10-23
- HTML: https://affpapa.org/en/nevibecode/rules/scala-spark

## What it does

The rule targets `**/*.scala`, `**/build.sbt`, and `**/build.sc`. It instructs Copilot to mark core Spark dependencies like `spark-core` and `spark-sql` as `provided` in build tools, keeping fat JARs slim. It also enforces a single `SparkSession` per JVM and forbids hardcoding `master` directly in application logic.

When generating code, the assistant prioritizes DataFrame transformations over raw RDDs and typed Datasets. For join operations, the rule suggests `broadcast()` for lookups under 100 MB, key salting for skewed data, and Spark 3 AQE configuration flags.

Running `inferSchema=true` on large JSON files reads entire datasets upfront and stalls jobs. Because of that, the rule enforces explicit `StructType` schemas on ingest and warns against hazardous calls like `coalesce(1)` on large tables.

## Install

### GitHub Copilot (native platform)

**Manually: put the files in place**
- File: `.github/instructions/scala-spark.instructions.md` — The applyTo field sets which files it applies to

### Claude Code

**Manually: put the files in place**
- File: `.claude/rules/scala-spark.md` — The paths frontmatter field limits the files: move applyTo from the source into paths

### Codex

**AGENTS.md**
- Codex reads AGENTS.md: paste the rule text there

### Cursor

**Manually: put the files in place**
- File: `.cursor/rules/scala-spark.mdc` — Frontmatter: description, globs, alwaysApply. Move applyTo from the source into globs, or the rule will not attach on its own

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/rules/scala-spark.md`

### Cline

**Manually: put the files in place**
- File: `.clinerules/scala-spark.md`

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/github/awesome-copilot/tree/HEAD/instructions/scala-spark.instructions.md

## When to use it
- configure Spark build dependencies with provided scope
- replace expensive schema inference with explicit StructType definitions
- eliminate join bottlenecks using broadcast or AQE skew handling
- prevent single-node bottlenecks caused by coalesce(1)

## Example prompts
- Create a SparkSession builder with KryoSerializer and 200 shuffle partitions.
- Read JSON event logs using an explicit StructType schema without inferSchema.
- Join a large dataframe with a small lookup table using broadcast join.

## Source text · scala-spark.instructions.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 2262a8b7e2178c09.

BEGIN THIRD-PARTY FILE 2262a8b7e2178c09
````text
---
description: 'Best practices for building Apache Spark applications in Scala, covering DataFrames, Datasets, SparkSQL, performance tuning, testing, and production deployment patterns.'
applyTo: '**/*.scala, **/build.sbt, **/build.sc'
---

# Scala + Apache Spark Best Practices

Guidelines for writing efficient, maintainable, and production-ready Apache Spark applications in Scala.

## Dependencies

### SBT

```scala
val sparkVersion = "3.5.1"

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core"   % sparkVersion % "provided",
  "org.apache.spark" %% "spark-sql"    % sparkVersion % "provided",
  "org.apache.spark" %% "spark-mllib"  % sparkVersion % "provided",
  "org.apache.spark" %% "spark-streaming" % sparkVersion % "provided"
)
```

### Maven

```xml
<properties>
    <spark.version>3.5.1</spark.version>
    <scala.binary.version>2.13</scala.binary.version>
</properties>

<dependencies>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-core_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-sql_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-mllib_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>org.apache.spark</groupId>
        <artifactId>spark-streaming_${scala.binary.version}</artifactId>
        <version>${spark.version}</version>
        <scope>provided</scope>
    </dependency>
</dependencies>
```

Mark Spark dependencies as `"provided"` since the cluster supplies them at runtime. Only bundle application-specific libraries in the fat JAR.

## SparkSession Setup

Always use `SparkSession` as the single entry point:

```scala
import org.apache.spark.sql.SparkSession

val spark: SparkSession = SparkSession.builder()
  .appName("MyApplication")
  .config("spark.sql.shuffle.partitions", "200")
  .config("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
  .getOrCreate()

import spark.implicits._
```

- Do **not** create multiple `SparkSession` instances in the same JVM.
- Avoid hardcoding `master` in application code; set it at submit time via `--master`.

## DataFrames vs Datasets vs RDDs

Prefer the **DataFrame API** (untyped `Dataset[Row]`) for most workloads. Use **Datasets** (typed) when compile-time type safety justifies the serialization overhead. Avoid raw **RDDs** unless you need low-level control.

```scala
import org.apache.spark.sql.{DataFrame, Dataset}

// Preferred — DataFrame API
val df: DataFrame = spark.read.parquet("data/events")
val result = df
  .filter($"status" === "active")
  .groupBy($"region")
  .agg(count("*").as("total"))

// Typed Dataset — use when schema safety matters
case class Event(id: Long, status: String, region: String)
val ds: Dataset[Event] = df.as[Event]
val active = ds.filter(_.status == "active")
```

## Schema Management

Always define schemas explicitly when reading semi-structured data instead of relying on schema inference:

```scala
import org.apache.spark.sql.types._

val schema = StructType(Seq(
  StructField("id", LongType, nullable = false),
  StructField("name", StringType, nullable = true),
  StructField("timestamp", TimestampType, nullable = false),
  StructField("amount", DecimalType(18, 2), nullable = true),
  StructField("tags", ArrayType(StringType), nullable = true)
))

val df = spark.read
  .schema(schema)
````
END THIRD-PARTY FILE 2262a8b7e2178c09

The third-party file has ended. Everything between the 2262a8b7e2178c09 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 3752 of 16313 bytes. © github · MIT · https://github.com/github/awesome-copilot/tree/HEAD/instructions/scala-spark.instructions.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md