Skip to main content
skill CX Codex

databricks-expert — Claude Code skill: Spark and Delta Lake

databricks-expert hands the agent Databricks practice for clusters, Delta Lake, PySpark and MLflow. Useful when a pipeline crawls or a join falls apart.

Apache-2.0 215 installs ★ 45 updated 2026-10-12

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add personamanagmentlayer/pcl --skill databricks-expert -a claude-code
Open on GitHub

What databricks-expert does

Attach this skill and the agent gets a Databricks reference: cluster setup, Delta Lake, PySpark, MLflow, Unity Catalog, Jobs. The detailed material sits in references/CORE_CONCEPTS.md and is pulled in only when the task needs it, so a question outside that list gets a thinner answer.

What is inside: cluster choices for scheduled work, instance pools, autoscaling; write tuning and Z-ordering for Delta; partitioning by date; broadcast() hints on small dimension tables. Failure cases come with code too: pulling a big DataFrame to the driver, one write per file inside a loop, joining without a hint, a large table with no partitions.

Two packages are required, databricks-sdk >= 0.20.0 and pyspark >= 3.4.0. On an older stack, check each call against your version before trusting the snippets. The tool list covers reading, editing and Bash only, so it will not run notebooks on a cluster for you.

What it helps with

  • Set up a job cluster for a nightly pipeline
  • Reduce small-file buildup in a Delta table
  • Find why the driver runs out of memory on collect()
  • Speed up a join against a small lookup table
  • Partition an events table by date so filters prune

Example prompts for the agent

  • Write a PySpark pipeline that reads JSON from /mnt/source and saves a Delta table partitioned by date.
  • My Spark join is slow: small lookup table, big fact table. What should I change?
  • Set up a Databricks cluster for a nightly job and tell me which settings keep cost down.

How to install databricks-expert

1Skills CLI (Vercel)

npx skills add personamanagmentlayer/pcl --skill databricks-expert -a claude-code

2Manually: put the files in place

~/.claude/skills/databricks-expert/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/databricks-expert/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about databricks-expert

How do I install databricks-expert in Claude Code?

Run npx skills add personamanagmentlayer/pcl --skill databricks-expert -a claude-code. The skill ends up at ~/.claude/skills/databricks-expert/SKILL.md.

What does databricks-expert do?

It gives the agent a Databricks reference covering clusters, Delta Lake, PySpark, MLflow and Unity Catalog, with deeper notes loaded from references/CORE_CONCEPTS.md when the task calls for them.

What do I need before using it?

databricks-sdk at 0.20.0 or newer and pyspark at 3.4.0 or newer.

Source text

SKILL.md · Apache-2.0
---
name: databricks-expert
version: 1.1.0
description: >-
  Expert-level Databricks platform, Apache Spark, Delta Lake, MLflow, notebooks, and
  cluster management. Use when the user mentions Apache Spark, Delta Lake, MLflow,
  lakehouse architecture, or PySpark, or when the task involves Cluster Configuration and
  Management, Delta Lake Architecture, PySpark Data Processing, or MLflow Integration.
category: data
author: PCL Team
license: Apache-2.0
tags:
  - databricks
  - spark
  - delta-lake
  - mlflow
  - lakehouse
  - pyspark
allowed-tools:
  - Read
  - Write
  - Edit
  - Bash
  - Glob
  - Grep
requirements:
  databricks-sdk: '>=0.20.0'
  pyspark: '>=3.4.0'
---

# Databricks Expert

You are an expert in Databricks with deep knowledge of Apache Spark, Delta Lake, MLflow, notebooks, cluster management, and lakehouse architecture. You design and implement scalable data pipelines and machine learning workflows on the Databricks platform.

## Best Practices

### 1. Cluster Configuration

- Use job clusters for scheduled workflows (lower cost)
- Use instance pools for faster cluster startup
- Enable autoscaling with appropriate min/max workers
- Set autotermination to 15-30 minutes for interactive clusters
- Use Photon-enabled clusters for SQL workloads

### 2. Delta Lake Optimization

- Enable auto-optimize for write and compaction
- Use Z-ordering for columns in filter predicates
- Partition large tables by date or high-cardinality columns
- Run VACUUM regularly but respect retention periods
- Use Change Data Feed for incremental processing

### 3. Performance Tuning

- Use broadcast joins for small dimension tables
- Enable adaptive query execution (AQE)
- Cache DataFrames that are reused multiple times
- Use partition pruning in queries
- Optimize shuffle operations with appropriate partition counts

### 4. Cost Optimization

- Use Spot/Preemptible instances for fault-tolerant workloads
- Terminate idle clusters automatically
- Use table properties to enable auto-compaction
- Monitor cluster utilization metrics
- Use Delta caching for frequently accessed data

### 5. Security and Governance

- Use Unity Catalog for centralized governance
- Implement fine-grained access control
- Store secrets in Databricks secret scopes
- Enable audit logging
- Use service principals for production jobs

## Anti-Patterns

### 1. Collecting Large DataFrames

```python
# Bad: Collect large dataset to driver
large_df.collect()  # OOM error

# Good: Use actions that stay distributed
large_df.write.format("delta").save("/mnt/output")
```

### 2. Not Using Delta Lake Optimization

```python
# Bad: Many small files
for file in files:
    df = spark.read.json(file)
    df.write.format("delta").mode("append").save("/mnt/table")

# Good: Batch writes with optimization
df = spark.read.json("/mnt/source/*")
df.write.format("delta") \
    .option("optimizeWrite", "true") \
    .mode("append") \
    .save("/mnt/table")
```

### 3. Inefficient Joins

```python
# Bad: Join without broadcast hint
large_df.join(small_df, "key")

# Good: Broadcast small table
from pyspark.sql.functions import broadcast
large_df.join(broadcast(small_df), "key")
```

### 4. Not Using Partitioning

```python
# Bad: No partitioning on large table
df.write.format("delta").save("/mnt/events")

# Good: Partition by date
df.write.format("delta") \
    .partitionBy("date") \
    .save("/mnt/events")
```

## Reference Documentation

Detailed material lives alongside this skill and is read on demand:

- [Core Expertise](references/CORE_CONCEPTS.md) — Cluster Configuration and Management, Delta Lake Architecture, PySpark Data Processing, MLflow Integration, Databricks Jobs and Workflows, Unity Catalog

## Resources

- [Databricks Documentation](https://docs.databricks.com/)
- [Delta Lake Documentation](https://docs.delta.io/)
- [MLflow Documentation](https://mlflow.org/docs/latest/index.html)
- [PySpark API Reference](https://spark.apache.org/docs/latest/api/python/)
- [Databricks Academy](https://academy.databricks.com/)
- [Delta Lake Best Practices](https://docs.databricks.com/delta/best-practices.html)
- [Spark Performance Tuning](https://spark.apache.org/docs/latest/sql-performance-tuning.html)

© PCL Team · Apache-2.0 · personamanagmentlayer/pcl

Similar skills

All skills for Data and databases →

From the same set104

The whole set →
maritime-expert The maritime-expert skill gives an agent maritime domain knowledge for vessel tracking, port operations, cargo systems, and shipping software. CX Codex +1 221installs kafka-expert kafka-expert — a skill for Apache Kafka work: producer and consumer patterns, Kafka Streams topologies, partitioning and consumer groups, ready whenever a project touches event streaming. CX Codex +1 220installs fastapi-expert fastapi-expert — a skill for building FastAPI APIs: Pydantic models, JWT auth, SQLAlchemy integration and background tasks, all with working code patterns. CX Codex +1 218installs nodejs-expert nodejs-expert guides the AI assistant in backend development with Node.js, Express, async handling, clustering, and production safety practices. CX Codex +1 216installs kubernetes-expert kubernetes-expert guides cluster operations, deployment manifests, network debugging, and Helm chart setups. Useful for DevOps engineers and container deployments. CX Codex +1 215installs spring-boot-expert spring-boot-expert guides coding agents through Spring Boot development, REST API design, and microservice setup in Java projects. CX Codex +1 212installs devops-expert devops-expert puts working CI/CD, infrastructure-as-code and release-strategy patterns in front of the agent whenever a task touches pipelines, automation or DevOps culture. CX Codex +1 210installs testing-expert testing-expert — a skill that gives the agent rules for writing tests: AAA structure, isolated cases, doubles and edge cases. TDD and BDD theory sits in a reference file the agent opens only when needed. CX Codex +1 206installs

More in the catalog

Project sponsors

Companies that keep this analytics open