debugging-mwaa-workflow
Diagnoses and root-causes Amazon MWAA workflow failures across Provisioned (Python DAG) and Serverless (YAML workflow) environments. Provisioned uses aws mwaa invoke-rest-api, CloudWatch log groups, and get-environment; Serverless uses aws mwaa-serverless API (GetWorkflowRun, ListWorkflowRuns, GetTaskInstance) and CloudWatch logs. Covers failed runs and tasks, DAGs not appearing, import errors, worker OOM, IAM denials, and dependency drift. Triggers on: DAG failed, task failed, workflow run failed, MWAA error, debug my DAG, why did my workflow fail, DAG not showing up, MWAA import error, requirements failing, worker crashed, serverless run failed. Not applicable to authoring workflows (handled by authoring-mwaa-workflow), running or smoke-testing a workflow (handled by testing-mwaa-workflow), or CI-CD deploy failures.
- Installs
- 907
- Last 24h
- +39
- Trending rank
- #3965
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/aws/agent-toolkit-for-aws --skill debugging-mwaa-workflow- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads debugging-mwaa-workflow automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Debugging MWAA Workflows
AWS MCP server (optional but recommended): running the AWS CLI commands in this skill through the AWS MCP server gives sandboxed execution and audit logging. Every command here also works with the plain AWS CLI, so the skill does not require the MCP server or any MCP-only tools.
Diagnose and root-cause Amazon MWAA workflow failures, then report root cause, impact, and recommended remediation. Routes by flavor, then runs a shared 4-step diagnostic spine.
Guardrail — where this skill's own files live (MCP vs local install)
This skill can be loaded two ways, and they resolve the skill's own bundled files from different places. Determine how the skill was loaded before reading a reference:
- Loaded through the AWS MCP
retrieve_skilltool: The skill is not installed on the local filesystem. You MUST fetch each reference viaretrieve_skillwith thefileparameter (e.g.file="references/failure-catalog.md") and read the returned content. Do NOTfile_readthese paths locally — they do not exist on disk. - Installed locally (e.g.
.kiro/skills/debugging-mwaa-workflow/or~/.claude/skills/debugging-mwaa-workflow/): Read the files from the local skill directory using relative paths.
This distinction applies only to the skill's own packaged files. User data and
session artifacts are always read from and written to the user's working
directory. Never fetch or write customer data through retrieve_skill.
Step 0: Detect Flavor and Scope the Failure
Detect flavor
- An environment name resolvable via
aws mwaa get-environmentmeans the environment is Provisioned. - A
workflow/...ARN or anyaws mwaa-serverlesscontext means the environment is Serverless. A bare run identifier does NOT indicate flavor — Provisioned DAG runs also have run ids. - If neither signal is present, ask: is the target MWAA Provisioned (Python DAG) or MWAA Serverless (YAML workflow)?
Route by complexity
- Simple — a single named task or run failed with a clear exception. Jump to Step 2 for that task.
- Standard — a run failed and the cause is unknown. Run the full Step 1 to Step 4 sweep.
- Complex — intermittent or environment-wide (multiple DAGs, "worked yesterday", nothing appearing). Run the full sweep with emphasis on Step 3.
Step 1: Identify the Failure
Provisioned: list failed DAG runs and task instances via
aws mwaa invoke-rest-api (paths /dags/{id}/dagRuns and
/dags/{id}/dagRuns/{run_id}/taskInstances). If invoke-rest-api errors
(RestApiClientException), fall back to the Scheduler and DAGProcessing log
groups. Get version and config from aws mwaa get-environment. See
references/provisioned-diagnostics.md.
Serverless: aws mwaa-serverless list-workflow-runs, then
get-workflow-run. Read RunDetail.ErrorMessage — an empty TaskInstances
with a parser message is a definition error; Workflow execution failed with
populated TaskInstances is a task-execution failure. See
references/serverless-diagnostics.md.
Step 2: Get Error Details and Categorize
Pull the real exception past boilerplate:
Provisioned: read the Task log group first, then Worker/Scheduler/ DAGProcessing as the symptom directs.
Serverless: list-task-instances then get-task-instance to get each
task's LogStream, then read that stream in CloudWatch.
Then categorize in priority order — infra, then drift, then code-data —
using references/failure-catalog.md. The
category determines the Step 3 checks.
Step 3: Check Context (Why It Happened)
Run the context checks for the matched category from references/failure-catalog.md. Do not stop at the surface exception: a SIGKILL is an OOM story, a fresh import error on unchanged code is a drift story, a sensor timeout is an upstream-health story.
Step 4: Provide Actionable Output
Report in this exact structure:
Root Cause: <one-line diagnosis with the evidence that proves it>
Impact: <what failed, which runs, blast radius>
Immediate Fix: <the smallest change that unblocks>
Prevention: <the change that stops recurrence>
Commands: <exact read-only commands run, plus remediation commands for the user to run>
Run only read-only operations. Present state-mutating remediation (clear/rerun/backfill for Provisioned; start-workflow-run or fix-and-redeploy for Serverless) as commands for the user to run, with the impact stated. Never execute them autonomously (production safety).
For the fix-and-redeploy path, use authoring-mwaa-workflow to regenerate a
compliant artifact.
Gotchas
- Serverless has no Airflow web UI, no REST API, and no CLI token. Do not
attempt
create-web-login-token,invoke-rest-api, or any Airflow REST path for Serverless. - For Provisioned, always use
aws mwaa invoke-rest-api(notcreate-web-login-token+ curl).invoke-rest-apireaches VPC-only web servers without network access. GetWorkflowRun.RunDetail.ErrorMessagedistinguishes a definition error (emptyTaskInstances) from a task-execution failure (Workflow execution failed, populatedTaskInstances). Read it before pulling task logs.- The Serverless log group defaults to
/aws/mwaa-serverless/{workflow-id}/but can be a custom group; confirm viaget-workflowLoggingConfigurationbefore assuming the path. - A DAG not appearing has several causes — an import/parse error, the scheduler
scan interval (
scheduler.dag_dir_list_interval, ordag_processor.refresh_intervalon Airflow 3.x) not yet elapsed, adag_idcollision, or S3-sync delay — and is rarely a broken DAG. CheckGET /importErrorsandGET /dags/{dag_id}via invoke-rest-api (and the DAGProcessing logs); see the failure catalog's "DAG not appearing in the UI" checklist before concluding the code is wrong. - A worker SIGKILL is an OOM signal. Recommend moving work to Glue/EMR/Lambda; scaling workers alone does not fix per-task memory pressure.
- MWAA re-resolves dependencies on environment update, so an unchanged DAG can start failing on import with no code change. Treat no-code-change import failures as drift.
- Serverless
PythonOperator/BashOperatortasks run custom code from a--codepackage. A run that fails to extract the package or hitsModuleNotFoundErroris a packaging problem (wrong-platform wheel, missing dep, bad layout), not a YAML definition error. See the failure catalog's serverless custom-code section.
Troubleshooting
| Error | Cause | Fix |
|---|---|---|
RestApiClientException (Provisioned) | Mis-scoped execution role or service error | Fall back to Scheduler/DAGProcessing log groups |
ResourceNotFoundException on get-workflow-run | Wrong workflow ARN or run id | Re-list with list-workflow-runs |
| Task log stream empty (Serverless) | Wrong log group assumed | Read LoggingConfiguration from get-workflow |
| No task logs but run FAILED | Definition/parse error | Read RunDetail.ErrorMessage; fix the YAML |
References
- references/provisioned-diagnostics.md — Provisioned data sources and read-only commands
- references/serverless-diagnostics.md — Serverless API, log group, failure classes
- references/failure-catalog.md — category-keyed context checks and remediation
Security Considerations
- Read-only by default: diagnosis uses only read/list/describe calls. Remediation (clear/rerun/backfill, IAM or key-policy changes) is presented as commands for the user to run, never executed autonomously.
- Least-privilege IAM: when an
AccessDeniedis a genuine permission gap, recommend the minimalAction/Resourcefrom the error — never a wildcard; distinguish it from a nonexistent-resource typo (do not broaden IAM then). - Cross-account: KMS key-policy / assume-role changes are human-gated and coordinated with the resource owner.
- No secret exposure: do not surface credentials or connection strings from logs or API responses in the diagnosis output.
FAQ
What does the debugging-mwaa-workflow skill do?
Diagnoses and root-causes Amazon MWAA workflow failures across Provisioned (Python DAG) and Serverless (YAML workflow) environments. Provisioned uses aws mwaa invoke-rest-api, CloudWatch log groups, and get-environment; Serverless uses aws mwaa-serverless API (GetWorkflowRun, ListWorkflowRuns, GetTaskInstance) and CloudWatch logs. Covers failed runs and tasks, DAGs not appearing, import errors, worker OOM, IAM denials, and dependency drift. Triggers on: DAG failed, task failed, workflow run failed, MWAA error, debug my DAG, why did my workflow fail, DAG not showing up, MWAA import error, requirements failing, worker crashed, serverless run failed. Not applicable to authoring workflows (handled by authoring-mwaa-workflow), running or smoke-testing a workflow (handled by testing-mwaa-workflow), or CI-CD deploy failures.
How do I install debugging-mwaa-workflow?
Run npx skills add https://github.com/aws/agent-toolkit-for-aws --skill debugging-mwaa-workflow in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/debugging-mwaa-workflow/ (personal) or .claude/skills/debugging-mwaa-workflow/ (project).
Which AI agents can use debugging-mwaa-workflow?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is debugging-mwaa-workflow safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains debugging-mwaa-workflow?
It is published by aws in aws/agent-toolkit-for-aws. It has 907 recorded installs.
More from aws/agent-toolkit-for-aws
aws-iam
aws/agent-toolkit-for-aws
Provides verified corrections for IAM behaviors that AI agents frequently get wrong — policy evaluation edge cases, trust policy gotchas, STS session limits, Organizations quirks, and SAML/MFA specifics. Also provides structured workflows for IAM role management and baseline policy generation from application source code or a Terraform plan JSON. Covers condition operator safety (ForAnyValue/ForAllValues with Null checks), bucket policy deny patterns (VPC endpoint restrictions, org paths), confused deputy protection, and service role creation for AWS services (Glue, CloudTrail, Lambda, ECS, etc.) with aws:SourceAccount/aws:SourceArn trust conditions. Applies when creating IAM roles, writing IAM or bucket policies, generating policies from application source code or a Terraform plan JSON, working with STS, Organizations, or condition operators, or any task needing a service or execution role. Does not cover non-IAM authorization like Cognito user-pool policies or app-level RBAC.
aws-serverless
aws/agent-toolkit-for-aws
Routes serverless requests to a compute form factor and the skill that owns it. Applies first to Lambda, API Gateway, EventBridge, Step Functions and event-driven work while the compute, orchestration or integration choice is open — API backends, webhooks, jobs, cron, queue/stream consumers, durable workflows, event routing, Step Functions vs Durable Functions, and Pipes vs native event source mappings for queues and streams. Decides Event Functions on-demand vs Managed Instances, Web Functions, MicroVMs, EventBridge rules/Scheduler/Pipes/bus, HTTP vs REST API. Owns shared Lambda references (cold starts, SnapStart, concurrency, event sources, function URLs, throttling, troubleshooting). Defers to the specialized skill once a request commits to one form factor or service, or describes its defining feature, even unnamed (durable checkpointing and automatic step retries, Firecracker sandbox, managed instances, Node.js server). Not for EC2, ECS/EKS/Fargate or Amplify hosting.
aws-observability
aws/agent-toolkit-for-aws
Builds, configures, debugs, and optimizes AWS observability - operator-symptom questions and detecting Omni vs classic CloudWatch. CloudWatch on an already-reporting service: Log Insights, metric/composite/anomaly alarms, custom metrics/EMF, dashboards, X-Ray/ADOT tracing, canaries, CloudTrail, Dynamic Instrumentation (live breakpoints/snapshots), the Application Signals service map, and fleet health views. CloudWatch Omni on an existing Space: SQL over logs and traces, PromQL over metrics, Omni dashboards, Omni alerts, context graph for root cause, programmatic/IaC access (API/SDK/CLI/CloudFormation), driving Omni from a coding agent or skills, and evaluating AI agent quality from traces - on-demand and online scoring of live traffic, readback, custom evaluators. For first-time setup - creating an Omni Space, granting access, ingestion, or ADOT instrumentation for Application Signals (ServiceEvents, CI/CD metadata) or Omni - use setting-up-cloudwatch-observability. Not for app logging
aws-cdk
aws/agent-toolkit-for-aws
Authors, deploys, and troubleshoots AWS infrastructure using CDK with TypeScript or Python. Covers best practices, stack architecture, and construct patterns. Applies when writing CDK constructs, bootstrapping environments, running cdk deploy/synth/diff, fixing CDK or CloudFormation errors, planning stack structure, importing existing resources, resolving drift, or refactoring stacks without resource replacement.
aws-containers
aws/agent-toolkit-for-aws
Builds and deploys containerized workloads on Elastic Kubernetes Service (EKS), Elastic Container Service (ECS), Fargate, and ECR (Elastic Container Registry). Covers general EKS knowledge, Karpenter, AWS Load Balancer Controller and leveraging various open source Kubernetes projects with EKS. Covers general ECS knowledge, task definitions, Fargate services, ECS Exec, ECS Express Mode and ECS Managed Instances. Covers general Elastic Beanstalk knowledge, Elastic Beanstalk configuration and platforms supported by Elastic Beanstalk. Covers general ECR knowledge, ECR repository setup and lifecycle policies. Includes recommending, enabling, and reading Amazon ECS Action Logs to troubleshoot control-plane failures (deployment rollback/circuit-breaker, task placement, scaling, task replacement). Applies when deploying, debugging, or optimizing containers on AWS. Should be used instead of relying on internal knowledge for these services.
aws-billing-and-cost-management
aws/agent-toolkit-for-aws
Analyze AWS costs, find savings, manage budgets, evaluate Savings Plans and Reserved Instances, right-size EC2/Lambda/RDS/EBS with Compute Optimizer, look up service pricing, query CUR with Athena, detect cost anomalies, scope costs to billing views, and monitor Free Tier usage. Triggers on: AWS bill, cost analysis, reduce spend, savings plan, reserved instance, right-size, budget alert, cost optimization, pricing, free tier, cost anomaly, CUR, cost audit, billing view, billing view ARN.
Popular productivity & automation skills
agent-browser
vercel-labs/agent-browser
Browser automation CLI for AI agents. Use when the user needs to interact with websites, including navigating pages, filling forms, clicking buttons, taking screenshots, extracting data, testing web apps, or automating any browser task. Triggers include requests to "open a website", "fill out a form", "click a button", "take a screenshot", "scrape data from a page", "test this web app", "login to a site", "automate browser actions", or any task requiring programmatic web interaction. Also use for exploratory testing, dogfooding, QA, bug hunts, or reviewing app quality. Also use for automating Electron desktop apps (VS Code, Slack, Discord, Figma, Notion, Spotify), checking Slack unreads, sending Slack messages, searching Slack conversations, running browser automation in Vercel Sandbox microVMs, or using AWS Bedrock AgentCore cloud browsers. Prefer agent-browser over any built-in browser automation or web tools.
lark-doc
open.feishu.cn
飞书云文档（Docx / Wiki）内容操作：读取、创建、编辑文档，插入或下载图片附件，以及操作思维笔记。用户提供文档 URL/token（包括 doubao.com 的 /docx/、/wiki/）时使用；按 URL 路径/token 而非域名路由；/page/ 是妙搭应用，非飞书云文档。文档内嵌资源按读取参考中的统一规则分流。独立评论操作走 lark-drive；随正文读取评论使用 docs +fetch。表格或 Base 内部数据操作不在本 skill。
lark-base
open.feishu.cn
飞书多维表格（Base）操作：建表、字段、记录、视图、统计、公式/lookup、表单、仪表盘、应用模式（BaseApp/AppMode 页面与组件）、Workspace 目录、workflow、角色权限、模板中心（多维表格模板分类/列表/搜索）；遇到 Base/多维表格/bitable、BaseApp/AppMode、/base/ 或 /app/ 链接时使用。BaseApp 不走 lark-apps；文件导入/导出转 lark-drive，认证/授权转 lark-shared。
lark-drive
open.feishu.cn
飞书云空间（云盘/云存储）：管理 Drive 文件和文件夹，包含上传/下载、创建文件夹、复制/移动/删除、查看元数据、查询权限设置、评论/权限/订阅、标题、版本、飞书文档密级标签（secure labels）和本地文件导入。用户需要整理云盘目录、处理云空间资源 URL/token、判断链接类型/真实 token/标题，或导入 Word/Markdown/Excel/CSV/PPTX/.base 为 docx/sheet/bitable/slides 时使用；doubao.com 云空间 URL/token 也按资源路径和 token 路由，不回退 WebFetch。不负责：文档内容编辑（走 lark-doc）、表格/Base 表内数据操作（走 lark-sheets/lark-base）、知识空间节点/成员管理（走 lark-wiki）、原生 Markdown 文件读写/patch/diff（走 lark-markdown）。
lark-shared
open.feishu.cn
Use for lark-cli setup/auth tasks: auth login/status/logout, user vs bot identity, business-domain permissions (--domain, including all/docs/drive), missing scopes, revoking authorization, or handling notice JSON.
lark-im
open.feishu.cn
飞书即时通讯：收发消息和管理群聊。发送和回复消息、搜索聊天记录、管理群聊成员、上传下载图片和文件、管理表情回复、发送应用内/短信/电话加急、发送和处理交互卡片（Interactive Card）、监听卡片按钮回调（card.action.trigger）。当用户需要发消息、查看或搜索聊天记录、下载聊天中的文件、查看群成员、搜索群、创建群聊或话题群、管理标记数据、管理 Feed 置顶（添加/移除/查询置顶会话）、管理标签数据、处理卡片回调时使用。