Skip to main content
skill CX Codex

databricks-2025 — Claude Code skill: what it does, how to install

ADF and Databricks 2025 integration patterns, linked service templates, and updated connector recipes.

MIT 168 installs ★ 66 updated 2026-06-18

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add JosiahSiegel/claude-plugin-marketplace --skill databricks-2025 -a claude-code
Open on GitHub

What databricks-2025 does

This skill supplies patterns for Azure Data Factory orchestrations targeting Databricks. It guides migration from legacy Notebook, Python, and JAR activities to the dedicated DatabricksJob activity type, ensuring proper use of jobParameters instead of outdated schemas.

Guidance covers authentication workflows using Azure Managed Identity, avoiding interactive MFA blockers enforced since October 2025. It also provides reference setups for ServiceNowV2, KeyPair security for Snowflake connectors, PostgreSQL SSL parameters, and Spark 3.3 configurations for Mapping Data Flows.

Cluster details must be excluded from Databricks linked services under this pattern because serverless compute orchestration belongs inside the Databricks job definition.

What it helps with

  • migrate legacy ADF notebook activities to the modern DatabricksJob activity
  • configure Managed Identity authentication between Azure Data Factory and Databricks
  • upgrade deprecated ServiceNow pipeline connectors to ServiceNowV2
  • set up Snowflake linked services with KeyPair credentials via Key Vault
  • tune Mapping Data Flows with Spark 3.3 dynamic partition pruning

Example prompts for the agent

  • Show me a DatabricksJob activity JSON snippet using jobParameters for an ADF pipeline
  • Create a Databricks linked service template configured for MSI authentication
  • Update this ADF copy activity to use ServiceNowV2Source

How to install databricks-2025

1Skills CLI (Vercel)

npx skills add JosiahSiegel/claude-plugin-marketplace --skill databricks-2025 -a claude-code

2Manually: put the files in place

~/.claude/skills/databricks-2025/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/databricks-2025/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about databricks-2025

How to install databricks-2025 in Claude Code?

Run npx skills add JosiahSiegel/claude-plugin-marketplace --skill databricks-2025 -a claude-code in your terminal. The configuration is written to ~/.claude/skills/databricks-2025/SKILL.md.

What does databricks-2025 do?

It gives the coding agent structural knowledge and JSON recipes for modern ADF pipelines orchestrating Databricks jobs and updated cloud data connectors.

Why does the linked service omit cluster settings?

The modern DatabricksJob activity delegates compute lifecycle directly to the Databricks workspace job, using serverless compute without ADF-level cluster properties.

Source text

SKILL.md · MIT
---
name: databricks-2025
description: |
  ADF + Databricks 2025 integration patterns.
  PROACTIVELY activate for: (1) Databricks Job activity in ADF, (2) DatabricksJob (preview) vs DatabricksNotebook activity, (3) ServiceNow V2 connector, (4) ADF managed identity authentication for Databricks, (5) Databricks serverless linked services, (6) Snowflake V2 connector, (7) Databricks job parameters and outputs, (8) MFA enforcement and authentication updates, (9) Unity Catalog integration, (10) Delta Live Tables orchestration from ADF.
  Provides: Databricks linked service templates (PAT, MSI, serverless), DatabricksJob activity examples, parameter passing recipes, and authentication migration guidance.
---

# Azure Data Factory Databricks Integration 2025

## Databricks Job Activity (Recommended 2025)

**CRITICAL UPDATE (2025):** The Databricks Job activity is now the **ONLY recommended method** for orchestrating Databricks in ADF. Microsoft strongly recommends migrating from legacy Notebook, Python, and JAR activities.

### Quick Reference

- **Activity type:** `DatabricksJob` (NOT `DatabricksSparkJob` or `DatabricksNotebook`)
- **Parameter property:** `jobParameters` (NOT `parameters`)
- **Linked service auth:** Managed Identity (`"authentication": "MSI"`) recommended
- **Cluster config:** Do NOT specify cluster properties in linked service; the Databricks Job controls compute

### Why Databricks Job Activity?

| Feature | Notebook Activity (Legacy) | Job Activity (2025) |
|---------|---------------------------|---------------------|
| Compute | Must configure cluster in linked service | Serverless by default |
| Workflow tasks | Single notebook | Multi-task DAGs (notebook, Python, SQL, DLT) |
| Retry | ADF-level only | Job-level + task-level |
| Repair runs | Not supported | Rerun failed tasks only |
| Git integration | Limited | Full Databricks Git support + DABs |
| Lineage | None | Built-in data lineage |
| If/Else logic | Must use ADF control flow | Native If/Else task types |

### Benefits Summary

1. **Serverless Execution** -- No cluster specification needed; automatic serverless compute with faster startup and lower costs
2. **Advanced Workflow Features** -- Run As, Task Values, Conditional Execution, AI/BI Tasks, Repair Runs, Notifications, Queuing
3. **Centralized Job Management** -- Jobs defined once in Databricks workspace; single source of truth with Git-backed versioning
4. **Cost Optimization** -- Serverless compute (pay only for execution), job clusters (auto-terminating), spot instance support

For complete JSON examples of Job activity, linked service, and pipeline configurations, see `references/databricks-job-examples.md`.

## Connectors and Enhancements (2025+)

### ServiceNow V2 Connector (RECOMMENDED - V1 End of Support)

**ServiceNow V1 connector is at End of Support. Migrate to V2 immediately.**

| Feature | V1 | V2 |
|---------|----|----|
| Linked service type | `ServiceNow` | `ServiceNowV2` |
| Source type | `ServiceNowSource` | `ServiceNowV2Source` |
| Query builder | Custom | Aligns with ServiceNow condition builder |
| Performance | Standard | Enhanced extraction |
| OData support | No | Yes |

**Migration steps:** Update linked service type to `ServiceNowV2`, update source type to `ServiceNowV2Source`, test queries in ServiceNow UI condition builder, adjust timeouts.

### Enhanced PostgreSQL Connector

Improved performance with 2025 SSL enhancements: `enableSsl: true`, `sslMode: "Require"`.

### Enhanced Snowflake Connector

Improved performance with KeyPair authentication support and Key Vault secret integration.

### Managed Identity for Azure Storage

New managed identity support for Azure Table Storage and Azure Files connectors (system-assigned and user-assigned).

### Mapping Data Flows - Spark 3.3

Spark 3.3 now powers Mapping Data Flows with 30% faster processing, Adaptive Query Execution (AQE), dynamic partition pruning, improved caching, and better column statistics.

### Azure DevOps Server 2022 Support

Git integration now supports on-premises Azure DevOps Server 2022 via the `hostName` property.

For complete JSON examples of all connectors, see `references/connector-examples.md`.

## Managed Identity 2025 Best Practices

### User-Assigned vs System-Assigned

| Scenario | Recommendation |
|----------|---------------|
| Single ADF, simple setup | System-assigned |
| Multiple data factories | User-assigned (shared identity) |
| Complex multi-environment | User-assigned |
| Granular permission control | User-assigned |
| Identity lifecycle independence | User-assigned |

Use ADF's centralized **Credentials** feature to consolidate Microsoft Entra ID-based credentials across multiple linked services.

### MFA Enforcement (Enforced Since October 2025)

Azure MFA is mandatory for all interactive user logins. Impact on ADF:
- Managed identities are **UNAFFECTED** -- no MFA required for service accounts
- Service principals with certificate auth are the recommended alternative to secrets
- All interactive user logins require MFA

### Principle of Least Privilege

| Resource | Source Role | Sink Role |
|----------|-----------|-----------|
| Storage Blob | `Storage Blob Data Reader` | `Storage Blob Data Contributor` |
| SQL Database | `db_datareader` | `db_datareader` + `db_datawriter` |
| Key Vault | `Get` secrets only | `Get` secrets only |

For complete managed identity JSON examples, see `references/connector-examples.md`.

## Best Practices (2025)

1. **Use Databricks Job Activity (MANDATORY)** -- Stop using Notebook, Python, JAR activities. Define workflows in Databricks workspace with serverless compute.

2. **Managed Identity Authentication (MANDATORY)** -- Use managed identities for ALL Azure resources. Leverage Credentials feature for consolidation. MFA-compliant since October 2025.

3. **Monitor Job Execution** -- Track Databricks Job run IDs from ADF output, log parameters for auditability, set up alerts for failures, leverage built-in lineage.

4. **Optimize Spark 3.3 (Data Flows)** -- Enable AQE, use 4-8 partitions per core, broadcast joins for small dimensions, dynamic partition pruning.

## Resources

- [Databricks Job Activity](https://learn.microsoft.com/azure/data-factory/transform-data-using-databricks-spark-job)
- [ADF Connectors](https://learn.microsoft.com/azure/data-factory/connector-overview)
- [Managed Identity Authentication](https://learn.microsoft.com/azure/data-factory/data-factory-service-identity)
- [Mapping Data Flows](https://learn.microsoft.com/azure/data-factory/concepts-data-flow-overview)

## Progressive Disclosure References

- **Databricks Job Examples**: `references/databricks-job-examples.md` - Complete JSON for Job activity, linked services, pipeline, and Databricks workspace job definition
- **Connector Examples**: `references/connector-examples.md` - Complete JSON for ServiceNow V2, PostgreSQL, Snowflake, Azure Storage MI, Mapping Data Flows, and Azure DevOps Server

© JosiahSiegel · MIT · JosiahSiegel/claude-plugin-marketplace

Similar skills

All skills for Data and databases →

From the same set96

The whole set →
powershell-2025-changes powershell-2025-changes is a Claude Code skill that lists PowerShell changes in 2025 and replacement cmdlets for scripts affected by them. CX Codex +1 174installs azure-well-architected-framework azure-well-architected-framework reviews Azure cloud architectures against the Well-Architected Framework pillars using proven Azure CLI templates. CX Codex +1 174installs windows-git-bash-paths The windows-git-bash-paths skill helps handle Windows paths in Git Bash when working with SSDT, SqlPackage, and DACPAC files. Use it when Git Bash rewrites command arguments and SqlPackage receives the wrong path. CX Codex +1 173installs index-strategies index-strategies is a Claude Code skill that designs, reviews, and tunes SQL Server indexes, from clustered and nonclustered to filtered and columnstore. CX Codex +1 172installs adf-validation-rules The adf-validation-rules skill checks Azure Data Factory activity nesting, pipeline limits, and common validation traps. Use it while designing or debugging ADF pipelines. CX Codex +1 168installs azure-openai-2025 azure-openai-2025 — a reference skill for Azure OpenAI Service: the 2025 model lineup, deployment through az, SKU and quota notes, Python SDK recipes. CX Codex +1 167installs github-actions-2025 github-actions-2025 guides your agent through modern GitHub Actions features, including 1 vCPU runners, immutable releases, and Node24 migrations. CX Codex +1 166installs defender-for-devops defender-for-devops is a Claude Code skill for adding security checks to Azure Pipelines. It activates for MSDO setup, code and dependency scans, container checks, and IaC analysis. CX Codex +1 166installs

More in the catalog

Project sponsors

Companies that keep this analytics open