databricks-2025
ADF + Databricks 2025 integration patterns. PROACTIVELY activate for: (1) Databricks Job activity in ADF, (2) DatabricksJob (preview) vs DatabricksNotebook activity, (3) ServiceNow V2 connector, (4) ADF managed identity authentication for Databricks, (5) Databricks serverless linked services, (6) Snowflake V2 connector, (7) Databricks job parameters and outputs, (8) MFA enforcement and authentication updates, (9) Unity Catalog integration, (10) Delta Live Tables orchestration from ADF. Provides: Databricks linked service templates (PAT, MSI, serverless), DatabricksJob activity examples, parameter passing recipes, and authentication migration guidance.
- Installs
- 164
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/josiahsiegel/claude-plugin-marketplace --skill databricks-2025- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads databricks-2025 automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Azure Data Factory Databricks Integration 2025
Databricks Job Activity (Recommended 2025)
CRITICAL UPDATE (2025): The Databricks Job activity is now the ONLY recommended method for orchestrating Databricks in ADF. Microsoft strongly recommends migrating from legacy Notebook, Python, and JAR activities.
Quick Reference
- Activity type:
DatabricksJob(NOTDatabricksSparkJoborDatabricksNotebook) - Parameter property:
jobParameters(NOTparameters) - Linked service auth: Managed Identity (
"authentication": "MSI") recommended - Cluster config: Do NOT specify cluster properties in linked service; the Databricks Job controls compute
Why Databricks Job Activity?
| Feature | Notebook Activity (Legacy) | Job Activity (2025) |
|---|---|---|
| Compute | Must configure cluster in linked service | Serverless by default |
| Workflow tasks | Single notebook | Multi-task DAGs (notebook, Python, SQL, DLT) |
| Retry | ADF-level only | Job-level + task-level |
| Repair runs | Not supported | Rerun failed tasks only |
| Git integration | Limited | Full Databricks Git support + DABs |
| Lineage | None | Built-in data lineage |
| If/Else logic | Must use ADF control flow | Native If/Else task types |
Benefits Summary
- Serverless Execution -- No cluster specification needed; automatic serverless compute with faster startup and lower costs
- Advanced Workflow Features -- Run As, Task Values, Conditional Execution, AI/BI Tasks, Repair Runs, Notifications, Queuing
- Centralized Job Management -- Jobs defined once in Databricks workspace; single source of truth with Git-backed versioning
- Cost Optimization -- Serverless compute (pay only for execution), job clusters (auto-terminating), spot instance support
For complete JSON examples of Job activity, linked service, and pipeline configurations, see references/databricks-job-examples.md.
Connectors and Enhancements (2025+)
ServiceNow V2 Connector (RECOMMENDED - V1 End of Support)
ServiceNow V1 connector is at End of Support. Migrate to V2 immediately.
| Feature | V1 | V2 |
|---|---|---|
| Linked service type | ServiceNow | ServiceNowV2 |
| Source type | ServiceNowSource | ServiceNowV2Source |
| Query builder | Custom | Aligns with ServiceNow condition builder |
| Performance | Standard | Enhanced extraction |
| OData support | No | Yes |
Migration steps: Update linked service type to ServiceNowV2, update source type to ServiceNowV2Source, test queries in ServiceNow UI condition builder, adjust timeouts.
Enhanced PostgreSQL Connector
Improved performance with 2025 SSL enhancements: enableSsl: true, sslMode: "Require".
Enhanced Snowflake Connector
Improved performance with KeyPair authentication support and Key Vault secret integration.
Managed Identity for Azure Storage
New managed identity support for Azure Table Storage and Azure Files connectors (system-assigned and user-assigned).
Mapping Data Flows - Spark 3.3
Spark 3.3 now powers Mapping Data Flows with 30% faster processing, Adaptive Query Execution (AQE), dynamic partition pruning, improved caching, and better column statistics.
Azure DevOps Server 2022 Support
Git integration now supports on-premises Azure DevOps Server 2022 via the hostName property.
For complete JSON examples of all connectors, see references/connector-examples.md.
Managed Identity 2025 Best Practices
User-Assigned vs System-Assigned
| Scenario | Recommendation |
|---|---|
| Single ADF, simple setup | System-assigned |
| Multiple data factories | User-assigned (shared identity) |
| Complex multi-environment | User-assigned |
| Granular permission control | User-assigned |
| Identity lifecycle independence | User-assigned |
Use ADF's centralized Credentials feature to consolidate Microsoft Entra ID-based credentials across multiple linked services.
MFA Enforcement (Enforced Since October 2025)
Azure MFA is mandatory for all interactive user logins. Impact on ADF:
- Managed identities are UNAFFECTED -- no MFA required for service accounts
- Service principals with certificate auth are the recommended alternative to secrets
- All interactive user logins require MFA
Principle of Least Privilege
| Resource | Source Role | Sink Role |
|---|---|---|
| Storage Blob | Storage Blob Data Reader | Storage Blob Data Contributor |
| SQL Database | db_datareader | db_datareader + db_datawriter |
| Key Vault | Get secrets only | Get secrets only |
For complete managed identity JSON examples, see references/connector-examples.md.
Best Practices (2025)
-
Use Databricks Job Activity (MANDATORY) -- Stop using Notebook, Python, JAR activities. Define workflows in Databricks workspace with serverless compute.
-
Managed Identity Authentication (MANDATORY) -- Use managed identities for ALL Azure resources. Leverage Credentials feature for consolidation. MFA-compliant since October 2025.
-
Monitor Job Execution -- Track Databricks Job run IDs from ADF output, log parameters for auditability, set up alerts for failures, leverage built-in lineage.
-
Optimize Spark 3.3 (Data Flows) -- Enable AQE, use 4-8 partitions per core, broadcast joins for small dimensions, dynamic partition pruning.
Resources
Progressive Disclosure References
- Databricks Job Examples:
references/databricks-job-examples.md- Complete JSON for Job activity, linked services, pipeline, and Databricks workspace job definition - Connector Examples:
references/connector-examples.md- Complete JSON for ServiceNow V2, PostgreSQL, Snowflake, Azure Storage MI, Mapping Data Flows, and Azure DevOps Server
FAQ
What does the databricks-2025 skill do?
ADF + Databricks 2025 integration patterns. PROACTIVELY activate for: (1) Databricks Job activity in ADF, (2) DatabricksJob (preview) vs DatabricksNotebook activity, (3) ServiceNow V2 connector, (4) ADF managed identity authentication for Databricks, (5) Databricks serverless linked services, (6) Snowflake V2 connector, (7) Databricks job parameters and outputs, (8) MFA enforcement and authentication updates, (9) Unity Catalog integration, (10) Delta Live Tables orchestration from ADF. Provides: Databricks linked service templates (PAT, MSI, serverless), DatabricksJob activity examples, parameter passing recipes, and authentication migration guidance.
How do I install databricks-2025?
Run npx skills add https://github.com/josiahsiegel/claude-plugin-marketplace --skill databricks-2025 in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/databricks-2025/ (personal) or .claude/skills/databricks-2025/ (project).
Which AI agents can use databricks-2025?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is databricks-2025 safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains databricks-2025?
It is published by josiahsiegel in josiahsiegel/claude-plugin-marketplace. It has 164 recorded installs.
More from josiahsiegel/claude-plugin-marketplace
tailwindcss-advanced-layouts
josiahsiegel/claude-plugin-marketplace
Tailwind CSS advanced layout techniques including CSS Grid and Flexbox patterns. PROACTIVELY activate for: (1) building complex layouts with CSS Grid, (2) grid-template-areas via Tailwind v4 arbitrary values, (3) responsive grid (grid-cols-, auto-fit, minmax), (4) Flexbox patterns (flex-1, flex-grow, gap), (5) sticky headers and footers, (6) holy grail layout, (7) masonry-style layouts, (8) container queries (@container) with Tailwind, (9) subgrid usage, (10) aspect-ratio utilities, (11) magazine-style multi-column layouts. Provides: Grid template recipes, container-query patterns, holy-grail templates, masonry alternatives, and aspect-ratio examples.
tailwindcss-animations
josiahsiegel/claude-plugin-marketplace
Tailwind CSS animations and transitions including built-in utilities and custom keyframes. PROACTIVELY activate for: (1) animate- utilities (spin, ping, pulse, bounce), (2) custom @keyframes via @theme animate, (3) transition utilities (transition, duration, ease), (4) transforms (scale, rotate, translate) with transition, (5) prefers-reduced-motion: reduce, (6) Framer Motion / Motion One integration with Tailwind, (7) CSS scroll-driven animations (animation-timeline), (8) view transitions API with Tailwind, (9) hover, focus, group-hover transitions, (10) staggered list animations. Provides: animation utility reference, custom keyframe examples, reduced-motion pattern, Framer Motion integration, and scroll-driven animation recipes.
tailwindcss-mobile-first
josiahsiegel/claude-plugin-marketplace
Mobile-first responsive design patterns with Tailwind CSS v4 (2025-2026). PROACTIVELY activate for: (1) mobile-first design with Tailwind breakpoints (sm:, md:, lg:, xl:, 2xl:), (2) responsive utility ordering (default = mobile, then breakpoints), (3) responsive typography (text-base sm:text-lg lg:text-xl), (4) responsive grids and flexbox, (5) hide/show across breakpoints (hidden md:block), (6) max- breakpoints for desktop-down overrides, (7) container queries (@container) for component-level responsiveness, (8) safe-area insets and notch handling, (9) landscape vs portrait orientation, (10) touch vs hover (hover: with @media). Provides: breakpoint reference, mobile-first patterns, container-query examples, safe-area recipes, and touch/hover handling.
powershell-master
josiahsiegel/claude-plugin-marketplace
Complete PowerShell expertise system across ALL platforms (Windows/Linux/macOS). PROACTIVELY activate for: (1) ANY PowerShell task (scripts/modules/cmdlets), (2) CI/CD automation (GitHub Actions/Azure DevOps/Bitbucket), (3) Cross-platform scripting, (4) Module discovery and management (PSGallery), (5) Azure/AWS/Microsoft 365 automation, (6) Script debugging and optimization, (7) Best practices and security. Provides: PowerShell 7+ features, popular module expertise (Az, Microsoft.Graph, PnP, AWS Tools), PSGallery integration, platform-specific guidance, CI/CD pipeline patterns, cmdlet syntax mastery, and production-ready scripting patterns. Ensures professional-grade, cross-platform PowerShell automation following industry standards.
docker-best-practices
josiahsiegel/claude-plugin-marketplace
Comprehensive Docker best practices for images, containers, and production deployments. PROACTIVELY activate for: (1) writing production-ready Dockerfiles, (2) multi-stage builds for lean images, (3) base image selection (distroless, alpine, scratch, slim), (4) layer caching and ordering, (5) .dockerignore hygiene, (6) HEALTHCHECK and graceful shutdown, (7) non-root user and least-privilege patterns, (8) ENV vs ARG and secret handling, (9) tag/digest pinning and reproducibility, (10) Compose for local development. Provides: Dockerfile templates by language, image-size optimization checklist, .dockerignore template, HEALTHCHECK patterns, and Compose dev-stack examples.
git-master
josiahsiegel/claude-plugin-marketplace
Complete Git expertise system for ALL git operations. PROACTIVELY activate for: (1) ANY Git task (basic/advanced/dangerous), (2) Repository management, (3) Branch strategies and workflows, (4) Conflict resolution, (5) History rewriting/recovery, (6) Platform-specific operations (GitHub/Azure DevOps/Bitbucket), (7) Advanced commands (rebase/cherry-pick/filter-repo). Provides: complete Git command reference, safety guardrails for destructive operations, platform best practices, workflow strategies, reflog recovery techniques, and expert guidance for even the most risky operations. Always asks user preference for automatic commits vs manual control.
Popular data & databases skills
supabase-postgres-best-practices
supabase/agent-skills
Postgres best practices maintained by Supabase, for Postgres running anywhere. Load this skill BEFORE writing or changing anything that lives in a Postgres database: creating or altering tables and columns (including choosing column types), schema design, migrations and declarative schema files, RLS policies and the tests that verify them, indexes, triggers, database functions, queues and scheduled jobs (pgcron, pgmq), vector/semantic search (pgvector), and restoring dumps (pgrestore) or importing data. Also load it when diagnosing slow queries, high CPU, timeouts, EXPLAIN plans, connection exhaustion, locking, bloat, or rows visible to the wrong user or tenant. This is not just a performance guide — schema, migration, security, and SQL authoring tasks need these rules too, even for a one-column change or a single query.
prisma-client-api
prisma/skills
Prisma Client API reference covering model queries, filters, operators, and client methods. Use when writing database queries, using CRUD operations, filtering data, or configuring Prisma Client. Triggers on "prisma query", "findMany", "create", "update", "delete", "$transaction".
prisma-database-setup
prisma/skills
Deprecated compatibility name for prisma-orm-setup. Use only when a user or existing integration explicitly invokes prisma-database-setup; otherwise select prisma-orm-setup.
prisma-cli
prisma/skills
Prisma ORM 7 CLI reference for existing Prisma 7 applications or explicitly selected Prisma 7 workflows. Covers prisma init, generate, migrate, db, dev, studio, and mcp. For setup, connection repair, or Prisma 8, use prisma-orm-setup; not the Prisma Platform CLI.
prisma-postgres
prisma/skills
Deprecated compatibility name for prisma-postgres-setup. Use only when a user or existing integration explicitly invokes prisma-postgres; otherwise select prisma-postgres-setup.
prisma-driver-adapter-implementation
prisma/skills
Required reference for Prisma ORM 7 SQL driver adapter work. Use when implementing or modifying adapters, adding database drivers, or touching SqlDriverAdapter, Transaction, savepoint, result mapping, or DriverAdapterError behavior. Covers current transaction lifecycle, optional savepoint hooks, original database-error preservation, and verification.