# databricks-pipelines

Guides agents in designing Lakeflow Spark Declarative Pipelines on Databricks with Python or SQL. It selects dataset types and configures Auto CDC before coding begins.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 987 installs · ★ 368
- Author: databricks
- License: LicenseRef-Unknown · policy=link-only · description only; the file stays with the author
- Source: https://github.com/databricks/databricks-agent-skills/tree/HEAD/skills/databricks-pipelines/SKILL.md
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/databricks-pipelines

## What it does

The skill outlines architecture decisions for Lakeflow Spark Declarative Pipelines, previously named Delta Live Tables. It enforces choices between Streaming Tables and Materialized Views. This avoids common validation errors, such as applying streaming aggregates to append-only tables without a batch read. The prompt directs intermediate logic into Temporary Views and merges streams using Append Flows instead of streaming unions.

Change data capture is handled through Auto CDC for both streams and batch snapshots. Snapshot processing in SQL via `AUTO CDC FROM SNAPSHOT` requires DBR 18.3 or newer, while older runtimes require the Python function `dp.create_auto_cdc_from_snapshot_flow`. Custom external destinations, including Kafka and Event Hubs sinks, remain exclusive to Python.

You need the `databricks CLI` at version 1.0.0 or higher, along with the parent `databricks-core` skill for authentication. Picking the wrong dataset type is costly. You cannot switch between Streaming Tables and Materialized Views in place, and a full refresh will not fix it. The existing table must be dropped manually.

## Requirements
- Works with: Databricks

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/databricks-pipelines/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/databricks-pipelines/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/databricks-pipelines/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/databricks-pipelines/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install databricks/databricks-agent-skills databricks-pipelines
```

2. **Manually: put the files in place**
- File: `.github/skills/databricks-pipelines/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/databricks/databricks-agent-skills --path skills/databricks-pipelines
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/databricks-pipelines/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/databricks/databricks-agent-skills/tree/HEAD/skills/databricks-pipelines/SKILL.md

## When to use it
- determine whether to create a Streaming Table or a Materialized View
- configure Auto CDC for event streams or recurring complete snapshots
- ingest files from cloud storage and Volumes using Auto Loader
- combine multiple streaming sources into one table using Append Flows
- route pipeline output to external Kafka or Event Hubs sinks in Python

## Example prompts
- Help me build a Lakeflow pipeline with Auto Loader ingesting files into a Streaming Table.
- Configure Auto CDC from snapshot in SQL for our daily customer data updates.
- How do I merge two streaming sources into one table using Databricks declarative pipelines?

## Similar
- [sql-queries](https://affpapa.org/en/nevibecode/skills/sql-queries.md) — skill · 4500 installs
- [migrating-dbt-project-across-platforms](https://affpapa.org/en/nevibecode/skills/migrating-dbt-project-across-platforms.md) — skill · 526 installs
- [neon-ai-gateway](https://affpapa.org/en/nevibecode/skills/neon-ai-gateway.md) — skill · 27700 installs
- [integration-eventing-cdc-configure](https://affpapa.org/en/nevibecode/skills/integration-eventing-cdc-configure.md) — skill · 5100 installs
- [amazon-keyspaces](https://affpapa.org/en/nevibecode/skills/amazon-keyspaces.md) — skill · 5000 installs
- [etl-connector-strategy](https://affpapa.org/en/nevibecode/skills/etl-connector-strategy.md) — skill · 8000 installs

From the same set: [databricks/databricks-agent-skills](https://affpapa.org/en/nevibecode/by/databricks/databricks-agent-skills.md) — 7 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md