databricks-pipelines — Claude Code skill: how to install
Guides agents in designing Lakeflow Spark Declarative Pipelines on Databricks with Python or SQL. It selects dataset types and configures Auto CDC before coding begins.
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a claude-codeWhat databricks-pipelines does
The skill outlines architecture decisions for Lakeflow Spark Declarative Pipelines, previously named Delta Live Tables. It enforces choices between Streaming Tables and Materialized Views. This avoids common validation errors, such as applying streaming aggregates to append-only tables without a batch read. The prompt directs intermediate logic into Temporary Views and merges streams using Append Flows instead of streaming unions.
Change data capture is handled through Auto CDC for both streams and batch snapshots. Snapshot processing in SQL via AUTO CDC FROM SNAPSHOT requires DBR 18.3 or newer, while older runtimes require the Python function dp.create_auto_cdc_from_snapshot_flow. Custom external destinations, including Kafka and Event Hubs sinks, remain exclusive to Python.
You need the databricks CLI at version 1.0.0 or higher, along with the parent databricks-core skill for authentication. Picking the wrong dataset type is costly. You cannot switch between Streaming Tables and Materialized Views in place, and a full refresh will not fix it. The existing table must be dropped manually.
What it helps with
- determine whether to create a Streaming Table or a Materialized View
- configure Auto CDC for event streams or recurring complete snapshots
- ingest files from cloud storage and Volumes using Auto Loader
- combine multiple streaming sources into one table using Append Flows
- route pipeline output to external Kafka or Event Hubs sinks in Python
Example prompts for the agent
- Help me build a Lakeflow pipeline with Auto Loader ingesting files into a Streaming Table.
- Configure Auto CDC from snapshot in SQL for our daily customer data updates.
- How do I merge two streaming sources into one table using Databricks declarative pipelines?
How to install databricks-pipelines
1Skills CLI (Vercel)
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a claude-code2Manually: put the files in place
~/.claude/skills/databricks-pipelines/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/databricks-pipelines/
1Skills CLI (Vercel)
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a codex2Manually: put the files in place
.agents/skills/databricks-pipelines/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a cursor2Manually: put the files in place
.cursor/skills/databricks-pipelines/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install databricks/databricks-agent-skills databricks-pipelines2Manually: put the files in place
.github/skills/databricks-pipelines/SKILL.md1Skills CLI (Vercel)
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a gemini-cli2Platform CLI
gemini skills install https://github.com/databricks/databricks-agent-skills --path skills/databricks-pipelinesManually: put the files in place
.devin/skills/databricks-pipelines/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about databricks-pipelines
How to install databricks-pipelines in Claude Code?
Run npx skills add databricks/databricks-agent-skills --skill databricks-pipelines -a claude-code.
What does databricks-pipelines do?
It provides decision trees and conventions for authoring Lakeflow Spark Declarative Pipelines on Databricks. It guides dataset selection, CDC configuration, and syntax validation for SQL and Python.
What are the prerequisites for databricks-pipelines?
It requires the databricks CLI at version 1.0.0 or higher. You should also invoke the parent databricks-core skill first for CLI credentials and workspace data discovery.
