QA Resilience

Quick Reference

When to Use This Skill

• retries, deadlines, hedging, breakers, bulkheads, and overload protection
• degraded-mode UX or API behavior
• service-mesh or gateway resilience policy
• chaos engineering, game days, DR drills, and fault injection
• release gates based on failure behavior, not only happy-path load tests

Route Elsewhere

• simple CRUD or single-process utilities -> ordinary error handling may be enough
• frontend-only failure behavior -> software-frontend
• service implementation details -> software-backend
• telemetry instrumentation -> qa-observability
• broader platform and incident operating model -> ops-devops-platform

Workflow

1. Identify the critical user journeys and the dependencies that can break them.
2. Define the contract per dependency:
   • timeout or deadline budget
   • retry ownership
   • breaker or outlier policy
   • concurrency and queue limits
   • degraded behavior if the dependency is unavailable
3. Decide where the policy lives:
   • app code
   • client library
   • mesh or gateway
4. Test in stages:
   • design review or checker output establishes policy coherence only
   • deterministic fault injection proves the response to a named controlled fault
   • staged chaos in non-production proves bounded behavior and recovery in that environment
   • narrow prod canary or game day proves only the observed target path and window, with guardrails
5. Define pass or fail signals:
   • burn rate
   • p95 or p99
   • fallback rate
   • breaker transitions
   • shed volume
   • recovery time
6. Record the injected fault, steady-state hypothesis, blast-radius limit, abort condition, observation window, recovery result, and dependencies simulated rather than observed.

Selective verification and hazard analysis

• Use formal methods when the question is whether a recovery/failover protocol preserves an invariant under concurrent faults, or when a counterexample is needed. Return the property, initial states, transitions/environment assumptions, checked bound, result/trace, and model-to-code limits. A finite invariant check does not establish liveness or recovery time. Skip for routine timeout/retry configuration and named fault tests whose behavior is already directly observable.
• Use safety engineering when retries, rollback, failover, operators, or other functioning controls can interact to cause a specified unacceptable loss, such as duplicate financial effects or destructive recovery. Return loss → hazard → unsafe control action and context → constraint → owner → verification evidence, including residual unknowns. Skip for ordinary availability or latency tuning without a harmful interaction question. Uptime and a hazard diagram do not prove safety.

Keep resilience policy, fault-injection execution, recovery evidence, and release recommendations with this skill; use the foundation only for the named gap.

Pattern Rules

• retries happen at one layer only
• deadlines come before retries
• hedging is only for idempotent or cancellation-safe reads
• overload handling must shed early instead of collapsing late
• overload tests pass on goodput plateauing past capacity, not on "503s appeared"; shed responses are retried only at the retry-owning layer, within budget, after Retry-After
• in regulated firms (EU DORA, UK FCA/PRA operational resilience), test each important business service against its impact tolerance, third parties included; counsel owns scoping — see disaster-recovery-testing.md
• readiness and liveness must stay bounded and shallow
• resilience behaviors belong in targeted checks; do not let rate-limit or degraded-mode coverage leak into unrelated happy-path suites
• prod experiments require blast-radius limits, abort criteria, dashboards, and owners

Failure Modes to Validate

• timeouts and deadline propagation
• retry storms and duplicate side effects
• partial dependency outages
• slow downstreams and long-tail latency
• queue buildup and connection-pool exhaustion
• one-bad-host behavior inside a pool
• degraded-mode responses and stale-data fallbacks
• rate-limit handling, Retry-After, and client backoff expectations
• visible state convergence after recovery or backend resets
• failover and failback behavior
• metastable failure: a self-sustaining feedback loop (retry amplification, cache-miss stampede, queue backlog, connection-pool churn) that does not self-resolve after the original trigger clears — see references/cascading-failure-prevention.md

Testing Ladder

Deterministic first

• inject latency, errors, timeouts, malformed payloads, and unavailable endpoints in controlled tests
• verify the intended control activates and the wrong controls do not

Chaos second

• start in non-production
• use a small blast radius and a fixed time window
• stop immediately on error-budget or customer-impact breach

DR and game days last

• validate RTO and RPO claims explicitly
• rehearse recovery ownership, not only technical failover

Operational Guardrails

• every experiment needs a stated hypothesis and steady-state metric
• every run should capture timestamps, targets, blast radius, and dashboard links
• telemetry fields for retries, breaker transitions, hedging, shedding, and fallback are part of the resilience contract
• releases should be gated on reliability behavior, not only resource usage

Anti-Patterns

• no timeouts
• retries at every hop
• fixed-interval or unbounded retries
• fixed per-call retry count with no system-wide retry budget (does not tighten as error rate rises)
• retries without idempotency
• hedging unsafe writes
• no bulkheads or queue bounds
• deep readiness or liveness checks
• silent degraded mode
• untested failover plans
• happy-path-only load testing

Scripts

Typical usage:

python scripts/resilience_checker.py assess --input data/sample-service-profile.json
python scripts/resilience_checker.py gaps --input data/sample-service-profile.json
python scripts/resilience_checker.py report --input data/sample-service-profile.json --output resilience-report.md

Supply actual JSON booleans and explicit hedging N/A rationale. The heuristic score describes declared coverage; it does not establish fault behavior or recovery. See scripts/README.md for the input format and scoring logic.

ASCII Flow

Resilience request
  -> Identify failure mode: slow, transient, sustained, overloaded, or degraded
  -> Set deadlines, retry budgets, isolation, and fallback ownership
  -> Add telemetry for saturation, errors, latency, and degraded behavior
  -> Validate with deterministic fault injection before chaos experiments
  -> Gate release on recovery evidence, SLO impact, and rollback path
  -> Document runbook actions and residual risk

Navigation

Foundation applied recipes

• references/reliability-theory-applied.md
• references/distributed-systems-applied.md

Core references

• references/circuit-breaker-patterns.md
• references/retry-patterns.md
• references/timeout-policies.md
• references/deadlines-hedging.md
• references/bulkhead-isolation.md
• references/load-shedding-backpressure.md
• references/gateway-mesh-resilience.md
• references/graceful-degradation.md
• references/health-check-patterns.md
• references/cascading-failure-prevention.md
• references/disaster-recovery-testing.md

Operational resources

• references/resilience-checklists.md
• references/chaos-engineering-guide.md
• references/chaos-tooling-recipes.md
• references/idempotency-key-design.md
• references/resilience-telemetry.md
• references/slo-as-code.md
• references/ai-llm-resilience-failure-modes.md
• assets/runbooks/resilience-runbook-template.md
• assets/testing/fault-injection-playbook.md
• assets/testing/template-resilience-test-plan.md
• data/sample-service-profile.json

Related Skills

• ops-devops-platform
• software-backend
• software-architecture-design
• qa-observability
• qa-debugging
• software-security-appsec
• data-sql-optimization
• dev-api-design

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.