databricks-spark-structured-streaming

Q: How do I install databricks-spark-structured-streaming?

Run npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-spark-structured-streaming in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/databricks-spark-structured-streaming/ (personal) or .claude/skills/databricks-spark-structured-streaming/ (project).

by databricksAI & Agents

Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.

Installs: 304
Last 24h: —
Trending rank: —
Security: Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-spark-structured-streaming

01
Install
Run the command in your project. The CLI asks which agents to install for.
02
Describe the task
Ask your agent for something this skill covers. It loads databricks-spark-structured-streaming automatically when relevant.
03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Spark Structured Streaming

Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.

Quick Start

from pyspark.sql.functions import col, from_json

# Basic Kafka to Delta streaming
df = (spark
    .readStream
    .format("kafka")
    .option("kafka.bootstrap.servers", "broker:9092")
    .option("subscribe", "topic")
    .load()
    .select(from_json(col("value").cast("string"), schema).alias("data"))
    .select("data.*")
)

df.writeStream \
    .format("delta") \
    .outputMode("append") \
    .option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
    .trigger(processingTime="30 seconds") \
    .start("/delta/target_table")

Core Patterns

Pattern	Description	Reference
Kafka Streaming	Kafka to Delta, Kafka to Kafka, Real-Time Mode	See references/kafka-streaming.md
Real-Time Mode (RTM)	Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), `transformWithState`, observability, error classes, delivery semantics	See references/real-time-mode.md
Lakebase Sink	Write streaming records into Lakebase Postgres with transactional upserts. Native `format("postgresql")` sink (DBR 18.3+) and manual `foreach` sink as a fallback	See references/lakebase-sink-python.md
Stream Joins	Stream-stream joins, stream-static joins	See references/stream-stream-joins.md, references/stream-static-joins.md
Multi-Sink Writes	Write to multiple tables, parallel merges	See references/multi-sink-writes.md
Merge Operations	MERGE performance, parallel merges, optimizations	See references/merge-operations.md

Configuration

Topic	Description	Reference
Checkpoints	Checkpoint management and best practices	See references/checkpoint-best-practices.md
Stateful Operations	Watermarks, state stores, RocksDB configuration	See references/stateful-operations.md
Trigger & Cost	Trigger selection, cost optimization, RTM	See references/trigger-and-cost-optimization.md

Best Practices

Topic	Description	Reference
Production Checklist	Comprehensive best practices	See references/streaming-best-practices.md

Production Checklist

Checkpoint location is persistent (UC volumes, not DBFS)
Unique checkpoint per stream
Fixed-size cluster (no autoscaling for streaming)
Monitoring configured (input rate, lag, batch duration)
Exactly-once verified (txnVersion/txnAppId)
Watermark configured for stateful operations
Left joins for stream-static (not inner)

FAQ

What does the databricks-spark-structured-streaming skill do?

How do I install databricks-spark-structured-streaming?

Run npx skills add https://github.com/databricks/databricks-agent-skills --skill databricks-spark-structured-streaming in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/databricks-spark-structured-streaming/ (personal) or .claude/skills/databricks-spark-structured-streaming/ (project).

Which AI agents can use databricks-spark-structured-streaming?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is databricks-spark-structured-streaming safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains databricks-spark-structured-streaming?

It is published by databricks in databricks/databricks-agent-skills. It has 304 recorded installs.