┌─────────────────────────────────────────────────────────┐
│  MINUTE 0-1: Acknowledge and Classify                   │
│  • Acknowledge the alert or report                      │
│  • Assign severity (SEV1-SEV4)                          │
│  • Designate incident commander                         │
├─────────────────────────────────────────────────────────┤
│  MINUTE 1-2: Assess Scope                               │
│  • Check health endpoints for all services              │
│  • Check error rate and latency dashboards              │
│  • Determine: which services are affected?              │
├─────────────────────────────────────────────────────────┤
│  MINUTE 2-3: Identify Recent Changes                    │
│  • Check: was there a recent deployment?                │
│  • Check: any infrastructure changes?                   │
│  • Check: any external dependency issues?               │
├─────────────────────────────────────────────────────────┤
│  MINUTE 3-4: Initial Communication                      │
│  • Post in #incidents channel                           │
│  • Update status page if SEV1/SEV2                      │
│  • Page additional responders if needed                 │
├─────────────────────────────────────────────────────────┤
│  MINUTE 4-5: Begin Investigation or Mitigate            │
│  • If recent deploy: consider immediate rollback        │
│  • If not deploy-related: begin diagnostic commands     │
│  • Start incident timeline log                          │
└─────────────────────────────────────────────────────────┘

./skills/incident-response/scripts/health-check-all-services.sh \
  --output-dir ./incident-triage/

INCIDENT DECLARED: [Title]
Severity: [SEV1/SEV2/SEV3/SEV4]
Commander: [Name]
Start time: [UTC timestamp]
Impact: [What users are experiencing]
Status: Investigating
Next update: [Time]

INCIDENT UPDATE: [Title]
Severity: [SEV level]
Duration: [Time since start]
Status: [Investigating/Identified/Mitigating/Resolved]
Current findings: [What we know]
Actions in progress: [What we are doing]
Next update: [Time]

# 1. Check application logs for errors
./skills/incident-response/scripts/fetch-logs.sh \
  --service backend \
  --since "15 minutes ago" \
  --output-dir ./incident-logs/

# 2. Check error rate from logs
docker logs app-backend --since 15m 2>&1 | grep -c "ERROR"

# 3. Check active connections and request patterns
curl -s http://localhost:8000/health/ready | jq .

# 4. Check if the issue is in a specific endpoint
docker logs app-backend --since 15m 2>&1 | \
  grep "ERROR" | \
  grep -oP '"path":"[^"]*"' | sort | uniq -c | sort -rn

# 5. Check Python process status
docker exec app-backend ps aux
docker exec app-backend python -c "import sys; print(sys.version)"

# 1. Check database connectivity
docker exec app-db pg_isready -U postgres

# 2. Check active connections (connection pool exhaustion?)
docker exec app-db psql -U postgres -d app_prod -c "
  SELECT count(*), state FROM pg_stat_activity
  GROUP BY state ORDER BY count DESC;
"

# 3. Check for long-running queries (locks, deadlocks?)
docker exec app-db psql -U postgres -d app_prod -c "
  SELECT pid, now() - pg_stat_activity.query_start AS duration,
         query, state
  FROM pg_stat_activity
  WHERE (now() - pg_stat_activity.query_start) > interval '30 seconds'
  AND state != 'idle'
  ORDER BY duration DESC;
"

# 4. Check for lock contention
docker exec app-db psql -U postgres -d app_prod -c "
  SELECT blocked_locks.pid AS blocked_pid,
         blocking_locks.pid AS blocking_pid,
         blocked_activity.query AS blocked_query
  FROM pg_catalog.pg_locks blocked_locks
  JOIN pg_catalog.pg_stat_activity blocked_activity
    ON blocked_activity.pid = blocked_locks.pid
  JOIN pg_catalog.pg_locks blocking_locks
    ON blocking_locks.locktype = blocked_locks.locktype
    AND blocking_locks.relation = blocked_locks.relation
    AND blocking_locks.pid != blocked_locks.pid
  JOIN pg_catalog.pg_stat_activity blocking_activity
    ON blocking_activity.pid = blocking_locks.pid
  WHERE NOT blocked_locks.granted;
"

# 5. Check disk space
docker exec app-db df -h /var/lib/postgresql/data

# 1. Check Redis connectivity
docker exec app-redis redis-cli ping

# 2. Check memory usage
docker exec app-redis redis-cli info memory | grep used_memory_human

# 3. Check connected clients
docker exec app-redis redis-cli info clients | grep connected_clients

# 4. Check slow log
docker exec app-redis redis-cli slowlog get 10

# 5. Check keyspace
docker exec app-redis redis-cli info keyspace

# 1. Check DNS resolution
nslookup api.example.com

# 2. Check SSL certificate expiry
echo | openssl s_client -servername api.example.com -connect api.example.com:443 2>/dev/null | \
  openssl x509 -noout -dates

# 3. Check container resource usage
docker stats --no-stream --format "table {{.Name}}\t{{.CPUPerc}}\t{{.MemUsage}}\t{{.NetIO}}"

# 4. Check disk space on host
df -h /

# 5. Check if dependent services are reachable
curl -sf https://external-api.example.com/health || echo "External API unreachable"

[Investigating] We are investigating reports of [issue description].
Users may experience [user-visible impact].
We will provide an update within [time].

[Identified] The issue has been identified as [brief description].
We are working on a fix. Estimated resolution: [time estimate].

[Resolved] The issue affecting [service] has been resolved.
The root cause was [brief description].
We apologize for the disruption and will publish a detailed post-mortem.

Why did users see 500 errors?
  -> Because the backend service returned errors to the load balancer.
Why did the backend service return errors?
  -> Because database connections timed out.
Why did database connections time out?
  -> Because the connection pool was exhausted.
Why was the connection pool exhausted?
  -> Because a new endpoint opened connections without releasing them.
Why were connections not released?
  -> Because the endpoint was missing the async context manager for sessions.

Root cause: Missing async context manager for database sessions in new endpoint.

python skills/incident-response/scripts/generate-incident-report.py \
  --title "Database connection pool exhaustion" \
  --severity SEV2 \
  --start-time "2024-01-15T14:30:00Z" \
  --end-time "2024-01-15T15:15:00Z" \
  --output-dir ./post-mortems/

# Runbook: [Service Name]

## Service Overview
- Purpose, dependencies, critical paths

## Common Issues
### Issue 1: [Description]
- **Symptoms:** [What you see]
- **Diagnosis:** [Commands to run]
- **Resolution:** [Steps to fix]

## Escalation
- On-call: #ops-oncall
- Service owner: @team-name

# Post-Mortem: [Incident Title]

## Summary
- **Date:** YYYY-MM-DD
- **Severity:** SEV1-4
- **Duration:** X hours
- **Impact:** [Users/revenue affected]

## Timeline
- HH:MM - [Event]

## Root Cause
[Technical explanation]

## Action Items
- [ ] [Preventive measure] - Owner: @name - Due: YYYY-MM-DD