senior-data-engineer
Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.
- Installs
- 193
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/borghei/claude-skills --skill senior-data-engineer- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads senior-data-engineer automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Senior Data Engineer
Generate pipeline configurations (Airflow, Prefect, Dagster), validate data quality with profiling and anomaly detection, and optimize SQL/Spark performance with actionable recommendations.
Core Capabilities
- Pipeline generation — Airflow/Prefect/Dagster DAG code for batch and incremental loads, with DAG validation.
- Data quality — schema validation, profiling, anomaly detection, data contracts, and Great Expectations suite generation.
- ETL/ELT optimization — SQL and Spark analysis, partition strategy, and query cost estimation per warehouse.
- Architecture decisions — batch vs streaming and warehouse vs lakehouse trade-off frameworks.
- Reliability patterns — incremental watermarks, dead letter queues, freshness checks, and schema-drift detection.
When to Use
- Designing a data architecture or choosing batch vs streaming / warehouse vs lakehouse.
- Building or generating Airflow/Spark/dbt pipelines.
- Adding data-quality checks or data contracts.
- Optimizing slow ETL/ELT queries or troubleshooting pipeline failures.
Clarify First
Before generating pipelines, confirm these inputs. If any is unknown or vague, ASK — do not assume:
- Orchestrator — Airflow / Prefect / Dagster (
--type; changes the generated DAG code) - Source, destination & load mode — systems involved and batch vs incremental (
--source/--destination/--mode; shapes the pipeline) - Data-quality expectations — the schema and contracts to enforce (drives the Great Expectations suite generation)
Stop rule: ask only the 2-3 that most change the output. If the user says "just draft it," proceed and list your assumptions at the top of the artifact.
Quick Start
# Generate an Airflow DAG for incremental PostgreSQL -> Snowflake
python scripts/pipeline_orchestrator.py generate \
--type airflow --source postgres --destination snowflake \
--tables orders,customers --mode incremental --schedule "0 5 * * *"
# Validate data quality against a schema
python scripts/data_quality_validator.py validate data.csv \
--schema schema.json --detect-anomalies --json
# Profile a dataset
python scripts/data_quality_validator.py profile data.csv --json
# Optimize a slow SQL query
python scripts/etl_performance_optimizer.py analyze-sql query.sql \
--warehouse snowflake --json
# Estimate query cost
python scripts/etl_performance_optimizer.py estimate-cost query.sql \
--warehouse bigquery --stats data_stats.json --json
Tools
| Tool | Subcommands | Purpose |
|---|---|---|
pipeline_orchestrator.py | generate, validate, template | Generate Airflow/Prefect/Dagster pipeline code, validate DAGs |
data_quality_validator.py | validate, profile, generate-suite, contract, schema | Schema validation, profiling, anomaly detection, Great Expectations |
etl_performance_optimizer.py | analyze-sql, analyze-spark, optimize-partition, estimate-cost, template | SQL/Spark optimization, partition strategy, cost estimation |
All subcommands support --json for machine-readable output and --output for file writing.
References
Load the reference that matches the task — keep this file lean and pull detail on demand:
- references/pipeline-workflows.md — the three end-to-end worked pipelines with code: batch ETL (PostgreSQL → dbt → Snowflake), real-time streaming (Kafka → Spark → Delta Lake), and the data-quality framework. Read when building a concrete pipeline.
- references/decisions-and-troubleshooting.md — the batch-vs-streaming and warehouse-vs-lakehouse decision frameworks, anti-patterns, and the troubleshooting table. Read when choosing an architecture or diagnosing a failure.
- references/data_pipeline_architecture.md — deep reference on pipeline architecture patterns. Read for architecture design depth.
- references/data_modeling_patterns.md — dimensional modeling and data-modeling patterns. Read when modeling marts and dimensions.
- references/dataops_best_practices.md — DataOps practices for CI/CD, testing, and operating pipelines. Read when operationalizing pipelines.
Integration Points
| Skill | Integration |
|---|---|
senior-data-scientist | Feature engineering consumes curated mart data |
senior-ml-engineer | ML pipelines depend on feature store tables |
senior-devops | CI/CD for dbt, Airflow deployment, container orchestration |
senior-architect | Architecture reviews for lakehouse vs warehouse decisions |
code-reviewer | Pipeline code reviews for DAGs, dbt models, Spark jobs |
FAQ
What does the senior-data-engineer skill do?
Data engineering for batch and streaming pipelines with Airflow, dbt, Spark, and Kafka. Use when designing data architectures, building pipelines, adding data-quality checks, optimizing ETL/ELT, or troubleshooting pipeline failures.
How do I install senior-data-engineer?
Run npx skills add https://github.com/borghei/claude-skills --skill senior-data-engineer in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/senior-data-engineer/ (personal) or .claude/skills/senior-data-engineer/ (project).
Which AI agents can use senior-data-engineer?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is senior-data-engineer safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains senior-data-engineer?
It is published by borghei in borghei/claude-skills. It has 193 recorded installs.
More from borghei/claude-skills
product-designer
borghei/claude-skills
Product design across UI/UX, design systems, prototyping, and user research. Use when creating user journey maps, building wireframes, defining design tokens and component systems, planning usability tests, or establishing design principles.
business-intelligence
borghei/claude-skills
Business intelligence across dashboard design, visualization, and reporting automation. Use when designing dashboards, building KPI frameworks, automating reports, creating data stories, or optimizing BI tool performance.
brand-strategist
borghei/claude-skills
Brand strategy covering positioning, identity, brand architecture, messaging frameworks, and governance. Use when developing brand positioning, creating messaging frameworks, auditing brand health, or building brand architecture.
data-analyst
borghei/claude-skills
Data analysis across SQL, visualization, statistics, and reporting. Use when writing SQL queries, building dashboards, performing cohort or funnel analysis, running hypothesis tests, or presenting data-driven recommendations.
Popular data & databases skills
supabase-postgres-best-practices
supabase/agent-skills
Postgres best practices maintained by Supabase, for Postgres running anywhere. Load this skill BEFORE writing or changing anything that lives in a Postgres database: creating or altering tables and columns (including choosing column types), schema design, migrations and declarative schema files, RLS policies and the tests that verify them, indexes, triggers, database functions, queues and scheduled jobs (pgcron, pgmq), vector/semantic search (pgvector), and restoring dumps (pgrestore) or importing data. Also load it when diagnosing slow queries, high CPU, timeouts, EXPLAIN plans, connection exhaustion, locking, bloat, or rows visible to the wrong user or tenant. This is not just a performance guide — schema, migration, security, and SQL authoring tasks need these rules too, even for a one-column change or a single query.
prisma-client-api
prisma/skills
Prisma Client API reference covering model queries, filters, operators, and client methods. Use when writing database queries, using CRUD operations, filtering data, or configuring Prisma Client. Triggers on "prisma query", "findMany", "create", "update", "delete", "$transaction".
prisma-database-setup
prisma/skills
Deprecated compatibility name for prisma-orm-setup. Use only when a user or existing integration explicitly invokes prisma-database-setup; otherwise select prisma-orm-setup.
prisma-cli
prisma/skills
Prisma ORM 7 CLI reference for existing Prisma 7 applications or explicitly selected Prisma 7 workflows. Covers prisma init, generate, migrate, db, dev, studio, and mcp. For setup, connection repair, or Prisma 8, use prisma-orm-setup; not the Prisma Platform CLI.
prisma-postgres
prisma/skills
Deprecated compatibility name for prisma-postgres-setup. Use only when a user or existing integration explicitly invokes prisma-postgres; otherwise select prisma-postgres-setup.
prisma-driver-adapter-implementation
prisma/skills
Required reference for Prisma ORM 7 SQL driver adapter work. Use when implementing or modifying adapters, adding database drivers, or touching SqlDriverAdapter, Transaction, savepoint, result mapping, or DriverAdapterError behavior. Covers current transaction lifecycle, optional savepoint hooks, original database-error preservation, and verification.