deploying-airflow
Deploys Airflow DAGs and projects. Use when deploying Airflow or answering anything about deployment - deploying DAGs/projects, pushing code, setting up CI/CD, deploying to production or deployment strategies for Airflow.
- Installs
- 790
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/astronomer/agents --skill deploying-airflow- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads deploying-airflow automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Deploying Airflow
This skill covers deploying Airflow DAGs and projects to production, whether using Astro (Astronomer's managed platform) or open-source Airflow on Docker Compose or Kubernetes.
Choosing a path: Astro is a good fit for managed operations and faster CI/CD. For open-source, use Docker Compose for dev and the Helm chart for production.
Astro (Astronomer)
Astro provides CLI commands and GitHub integration for deploying Airflow projects.
Deploy Commands
| Command | What It Does |
|---|---|
astro deploy | Full project deploy — builds Docker image and deploys DAGs |
astro deploy --dags | DAG-only deploy — pushes only DAG files (fast, no image build) |
astro deploy --image | Image-only deploy — pushes only the Docker image (for multi-repo CI/CD) |
astro deploy --dbt | dbt project deploy — deploys a dbt project to run alongside Airflow |
Full Project Deploy
Builds a Docker image from your Astro project and deploys everything (DAGs, plugins, requirements, packages):
astro deploy
Use this when you've changed requirements.txt, Dockerfile, packages.txt, plugins, or any non-DAG file.
DAG-Only Deploy
Pushes only files in the dags/ directory without rebuilding the Docker image:
astro deploy --dags
This is significantly faster than a full deploy since it skips the image build. Use this when you've only changed DAG files and haven't modified dependencies or configuration.
Image-Only Deploy
Pushes only the Docker image without updating DAGs:
astro deploy --image
This is useful in multi-repo setups where DAGs are deployed separately from the image, or in CI/CD pipelines that manage image and DAG deploys independently.
dbt Project Deploy
Deploys a dbt project to run with Cosmos on an Astro deployment:
astro deploy --dbt
GitHub Integration
Astro supports branch-to-deployment mapping for automated deploys:
- Map branches to specific deployments (e.g.,
main-> production,develop-> staging) - Pushes to mapped branches trigger automatic deploys
- Supports DAG-only deploys on merge for faster iteration
Configure this in the Astro UI under Deployment Settings > CI/CD.
CI/CD Patterns
Common CI/CD strategies on Astro:
- DAG-only on feature branches: Use
astro deploy --dagsfor fast iteration during development - Full deploy on main: Use
astro deployon merge to main for production releases - Separate image and DAG pipelines: Use
--imageand--dagsin separate CI jobs for independent release cycles
Deploy Queue
When multiple deploys are triggered in quick succession, Astro processes them sequentially in a deploy queue. Each deploy completes before the next one starts.
Reference
Open-Source: Docker Compose
Deploy Airflow using the official Docker Compose setup. This is recommended for learning and exploration — for production, use Kubernetes with the Helm chart (see below).
Prerequisites
- Docker and Docker Compose v2.14.0+
- The official
apache/airflowDocker image
Quick Start
Download the official Airflow 3 Docker Compose file:
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/stable/docker-compose.yaml'
This sets up the full Airflow 3 architecture:
| Service | Purpose |
|---|---|
airflow-apiserver | REST API and UI (port 8080) |
airflow-scheduler | Schedules DAG runs |
airflow-dag-processor | Parses and processes DAG files |
airflow-worker | Executes tasks (CeleryExecutor) |
airflow-triggerer | Handles deferrable/async tasks |
postgres | Metadata database |
redis | Celery message broker |
Minimal Setup
For a simpler setup with LocalExecutor (no Celery/Redis), create a docker-compose.yaml:
x-airflow-common: &airflow-common
image: apache/airflow:3 # Use the latest Airflow 3.x release
environment: &airflow-common-env
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__DATABASE__SQL_ALCHEMY_CONN: postgresql+psycopg2://airflow:airflow@postgres/airflow
AIRFLOW__CORE__LOAD_EXAMPLES: 'false'
AIRFLOW__CORE__DAGS_FOLDER: /opt/airflow/dags
volumes:
- ./dags:/opt/airflow/dags
- ./logs:/opt/airflow/logs
- ./plugins:/opt/airflow/plugins
depends_on:
postgres:
condition: service_healthy
services:
postgres:
image: postgres:16
environment:
POSTGRES_USER: airflow
POSTGRES_PASSWORD: airflow
POSTGRES_DB: airflow
volumes:
- postgres-db-volume:/var/lib/postgresql/data
healthcheck:
test: ["CMD", "pg_isready", "-U", "airflow"]
interval: 10s
retries: 5
start_period: 5s
airflow-init:
<<: *airflow-common
entrypoint: /bin/bash
command:
- -c
- |
airflow db migrate
airflow users create \
--username admin \
--firstname Admin \
--lastname User \
--role Admin \
--email admin@example.com \
--password admin
depends_on:
postgres:
condition: service_healthy
airflow-apiserver:
<<: *airflow-common
command: airflow api-server
ports:
- "8080:8080"
healthcheck:
test: ["CMD", "curl", "--fail", "http://localhost:8080/health"]
interval: 30s
timeout: 10s
retries: 5
start_period: 30s
airflow-scheduler:
<<: *airflow-common
command: airflow scheduler
airflow-dag-processor:
<<: *airflow-common
command: airflow dag-processor
airflow-triggerer:
<<: *airflow-common
command: airflow triggerer
volumes:
postgres-db-volume:
Airflow 3 architecture note: The webserver has been replaced by the API server (
airflow api-server), and the DAG processor now runs as a standalone process separate from the scheduler.
Common Operations
# Start all services
docker compose up -d
# Stop all services
docker compose down
# View logs
docker compose logs -f airflow-scheduler
# Restart after requirements change
docker compose down && docker compose up -d --build
# Run a one-off Airflow CLI command
docker compose exec airflow-apiserver airflow dags list
Installing Python Packages
Add packages to requirements.txt and rebuild:
# Add to requirements.txt, then:
docker compose down
docker compose up -d --build
Or use a custom Dockerfile:
FROM apache/airflow:3 # Pin to a specific version (e.g., 3.1.7) for reproducibility
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
Update docker-compose.yaml to build from the Dockerfile:
x-airflow-common: &airflow-common
build:
context: .
dockerfile: Dockerfile
# ... rest of config
Environment Variables
Configure Airflow settings via environment variables in docker-compose.yaml:
environment:
# Core settings
AIRFLOW__CORE__EXECUTOR: LocalExecutor
AIRFLOW__CORE__PARALLELISM: 32
AIRFLOW__CORE__MAX_ACTIVE_TASKS_PER_DAG: 16
# Email
AIRFLOW__EMAIL__EMAIL_BACKEND: airflow.utils.email.send_email_smtp
AIRFLOW__SMTP__SMTP_HOST: smtp.example.com
# Connections (as URI)
AIRFLOW_CONN_MY_DB: postgresql://user:pass@host:5432/db
Open-Source: Kubernetes (Helm Chart)
Deploy Airflow on Kubernetes using the official Apache Airflow Helm chart.
Prerequisites
- A Kubernetes cluster
kubectlconfiguredhelminstalled
Installation
# Add the Airflow Helm repo
helm repo add apache-airflow https://airflow.apache.org
helm repo update
# Install with default values
helm install airflow apache-airflow/airflow \
--namespace airflow \
--create-namespace
# Install with custom values
helm install airflow apache-airflow/airflow \
--namespace airflow \
--create-namespace \
-f values.yaml
Key values.yaml Configuration
# Executor type
executor: KubernetesExecutor # or CeleryExecutor, LocalExecutor
# Airflow image (pin to your desired version)
defaultAirflowRepository: apache/airflow
defaultAirflowTag: "3" # Or pin: "3.1.7"
# Git-sync for DAGs (recommended for production)
dags:
gitSync:
enabled: true
repo: https://github.com/your-org/your-dags.git
branch: main
subPath: dags
wait: 60 # seconds between syncs
# API server (replaces webserver in Airflow 3)
apiServer:
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "500m"
memory: "1Gi"
replicas: 1
# Scheduler
scheduler:
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "1000m"
memory: "2Gi"
# Standalone DAG processor
dagProcessor:
enabled: true
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "500m"
memory: "1Gi"
# Triggerer (for deferrable tasks)
triggerer:
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "500m"
memory: "1Gi"
# Worker resources (CeleryExecutor only)
workers:
resources:
requests:
cpu: "500m"
memory: "1Gi"
limits:
cpu: "2000m"
memory: "4Gi"
replicas: 2
# Log persistence
logs:
persistence:
enabled: true
size: 10Gi
# PostgreSQL (built-in)
postgresql:
enabled: true
# Or use an external database
# postgresql:
# enabled: false
# data:
# metadataConnection:
# user: airflow
# pass: airflow
# host: your-rds-host.amazonaws.com
# port: 5432
# db: airflow
Upgrading
# Upgrade with new values
helm upgrade airflow apache-airflow/airflow \
--namespace airflow \
-f values.yaml
# Upgrade to a new Airflow version
helm upgrade airflow apache-airflow/airflow \
--namespace airflow \
--set defaultAirflowTag="<version>"
DAG Deployment Strategies on Kubernetes
- Git-sync (recommended): DAGs are synced from a Git repository automatically
- Persistent Volume: Mount a shared PV containing DAGs
- Baked into image: Include DAGs in a custom Docker image
Useful Commands
# Check pod status
kubectl get pods -n airflow
# View scheduler logs
kubectl logs -f deployment/airflow-scheduler -n airflow
# Port-forward the API server
kubectl port-forward svc/airflow-apiserver 8080:8080 -n airflow
# Run a one-off CLI command
kubectl exec -it deployment/airflow-scheduler -n airflow -- airflow dags list
Related Skills
- setting-up-astro-project: For initializing a new Astro project
- managing-astro-local-env: For local development with
astro dev - authoring-dags: For writing DAGs before deployment
- testing-dags: For testing DAGs before deployment
FAQ
What does the deploying-airflow skill do?
Deploys Airflow DAGs and projects. Use when deploying Airflow or answering anything about deployment - deploying DAGs/projects, pushing code, setting up CI/CD, deploying to production or deployment strategies for Airflow.
How do I install deploying-airflow?
Run npx skills add https://github.com/astronomer/agents --skill deploying-airflow in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/deploying-airflow/ (personal) or .claude/skills/deploying-airflow/ (project).
Which AI agents can use deploying-airflow?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is deploying-airflow safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains deploying-airflow?
It is published by astronomer in astronomer/agents, and it is an official skill from the maker of the technology it covers. It has 790 recorded installs.
More from astronomer/agents
airflow
astronomer/agents
Queries, manages, and troubleshoots Apache Airflow using the af CLI. Use when working with anything related to Airflow - a DAG, a DAG run, a task log, an import or parse error, a broken DAG, or any Airflow operation. Covers listing and triggering DAGs, retrying runs, reading task logs, diagnosing failures, debugging import and parse errors, checking connections, variables and pools, exploring the REST API, and monitoring health (for example "trigger a pipeline", "retry a run", "list connections", "check Airflow health", "why did my DAG fail"). This is the entrypoint that routes to sibling skills for authoring, testing, deploying, and migrating Airflow 2 to 3. Not for warehouse/SQL analytics on Airflow metadata tables (use analyzing-data); for deep root-cause reports use debugging-dags or airflow-investigation.
managing-astro-local-env
astronomer/agents
Manage local Airflow environment with Astro CLI (Docker and standalone modes). Use when the user wants to start, stop, or restart Airflow, view logs, query the Airflow API, troubleshoot, or fix environment issues. For project setup, see setting-up-astro-project.
analyzing-data
astronomer/agents
Queries the data warehouse with SQL and answers business questions about data. Use when answering anything that needs warehouse data - counts, metrics, trends, aggregations, joins across tables, data lookups, or ad-hoc SQL analysis (for example "who uses X", "how many Y", "show me Z", "find customers", "what is the count").
authoring-dags
astronomer/agents
Workflow and best practices for writing Apache Airflow DAGs. Use when creating a new DAG, write pipeline code, handling questions about DAG patterns and conventions or extending an existing DAG with a follow-up/downstream task. ANY request shaped like 'add a DAG named X', 'write a pipeline', 'add a task that runs after Y', or 'extend the DAG'. For testing and debugging DAGs, see the testing-dags skill.
debugging-dags
astronomer/agents
Comprehensive DAG failure diagnosis and root-cause analysis with structured investigation and prevention recommendations. Use when deep failure investigation is needed, a DAG fails to import/parse or 'airflow dags list' errors on a file; a task or run is failing and must be diagnosed and fixed; requests like 'why did X fail', 'my dag keeps failing — find and fix it', or fixing a broken DAG so it loads cleanly. For simple 'why did it fail / show logs', the airflow skill handles it directly.
testing-dags
astronomer/agents
Complex DAG testing workflows with debugging and fixing cycles. Use for multi-step testing requests like "test this dag and fix it if it fails", "test and debug", "run the pipeline and troubleshoot issues". For simple test requests ("test dag", "run dag"), the airflow entrypoint skill handles it directly. This skill is for iterative test-debug-fix cycles.
Popular devops & cloud skills
microsoft-foundry
microsoft/azure-skills
Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent create, managed harness agent create, GitHub Copilot harness agent, create agent, update agent, add tool to agent, invoke agent, agent insights, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure
azure-diagnostics
microsoft/azure-skills
Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.
azure-prepare
microsoft/azure-skills
Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.
azure-ai
microsoft/azure-skills
Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.
azure-deploy
microsoft/azure-skills
Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/deployment-plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.
azure-validate
microsoft/azure-skills
Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity permissions, and prerequisites before deploying. WHEN: validate my app, check deployment readiness, run preflight checks, verify configuration, check if ready to deploy, validate azure.yaml, validate Bicep, test before deploying, troubleshoot deployment errors, validate Azure Functions, validate function app, validate serverless deployment, verify RBAC roles, check role assignments, review managed identity permissions, what-if analysis, validate Container Apps deployment.