# data-engineering-patterns-fabric-databricks

data-engineering-patterns-fabric-databricks — a skill with 600+ data engineering patterns for Azure Databricks, Microsoft Fabric and PySpark: lakehouse architecture, Delta Lake, pipelines, cluster optimization.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 1600 installs · ★ 5 repo stars
- Author: reason-machines
- License: not stated · policy=link-only · description only; the file stays with the author
- Source: https://github.com/reason-machines/data-skills/tree/HEAD/skills/data-engineering-patterns-fabric-databricks/SKILL.md
- Updated: 2026-10-09
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/data-engineering-patterns-fabric-databricks

## What it does

The skill gives the agent access to field-tested patterns for building data pipelines in the Microsoft and Azure ecosystem. Inside are 12 PDF books organized by topic: pipelines and Data Factory in Fabric, lakehouse and PySpark, warehouse and SQL, Power BI, architecture patterns; clusters and compute in Databricks, Delta Lake, orchestration, streaming and Auto Loader, Unity Catalog, Databricks SQL and Photon, platform architecture and cost management.

Each pattern includes a problem statement, code sample and application context: incremental load with watermarks, data merge via `MERGE`, Delta table optimization with `ZORDER`, cluster autoscaling setup, table versioning through time travel. The agent pulls the pattern from the book and adapts it to the project context.

The skill requires manual repo cloning — PDFs live in `Fabric Patterns/`, `Databricks Patterns/` and `PySpark/` folders. After install, the agent references files by book name.

## Requirements
- Works with: Databricks, Azure

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-patterns-fabric-databricks -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/data-engineering-patterns-fabric-databricks/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/data-engineering-patterns-fabric-databricks/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-patterns-fabric-databricks -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/data-engineering-patterns-fabric-databricks/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-patterns-fabric-databricks -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/data-engineering-patterns-fabric-databricks/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install reason-machines/data-skills skills/data-engineering-patterns-fabric-databricks
```

2. **Manually: put the files in place**
- File: `.github/skills/data-engineering-patterns-fabric-databricks/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-patterns-fabric-databricks -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/reason-machines/data-skills --path skills/data-engineering-patterns-fabric-databricks
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/data-engineering-patterns-fabric-databricks/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add reason-machines/data-skills --skill data-engineering-patterns-fabric-databricks -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/reason-machines/data-skills/tree/HEAD/skills/data-engineering-patterns-fabric-databricks/SKILL.md

## When to use it
- build an incremental pipeline with watermarks in Fabric
- merge new records into a Delta table via `MERGE`
- configure Databricks cluster autoscaling with Spot fallback
- optimize Delta Lake with `ZORDER` and vacuum
- organize metadata in Unity Catalog

## Example prompts
- Show incremental load patterns for Fabric Data Factory
- How do I configure Databricks cluster autoscaling for production workloads?
- Give me a Delta Lake merge pattern with update and insert

## Similar
- [azure-databricks-platform-architect](https://affpapa.org/en/nevibecode/agents/azure-databricks-platform-architect.md) — subagent · ★ 6505 repo stars
- [databricks-vector-search](https://affpapa.org/en/nevibecode/skills/databricks-vector-search.md) — skill · 453 installs
- [databricks-docs](https://affpapa.org/en/nevibecode/skills/databricks-docs.md) — skill · 376 installs
- [databricks-expert](https://affpapa.org/en/nevibecode/skills/databricks-expert.md) — skill · 215 installs
- [oracle-ai-data-platform-workbench-databricks-migrator](https://affpapa.org/en/nevibecode/plugins/oracle-ai-data-platform-workbench-databricks-migrator.md) — plugin · ★ 62 repo stars
- [adf-ml-analytics](https://affpapa.org/en/nevibecode/skills/adf-ml-analytics.md) — skill · 113 installs

From the same set: [reason-machines/data-skills](https://affpapa.org/en/nevibecode/by/reason-machines/data-skills.md) — 82 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
