MCPInstall

observability-monitoring

by manutejDevOps & Cloud

Comprehensive observability and monitoring skill covering Prometheus, Grafana, metrics collection, alerting, exporters, PromQL, and production monitoring patterns for distributed systems and cloud-native applications

Installs
455
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/manutej/luxor-claude-marketplace --skill observability-monitoring
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads observability-monitoring automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Observability & Monitoring

A comprehensive skill for implementing production-grade observability and monitoring using Prometheus, Grafana, and the wider cloud-native monitoring ecosystem. This skill covers metrics collection, time-series analysis, alerting, visualization, and operational excellence patterns.

When to Use This Skill

Use this skill when:

  • Setting up monitoring for production systems and applications
  • Implementing metrics collection and observability for microservices
  • Creating dashboards and visualizations for system health monitoring
  • Defining alerting rules and incident response automation
  • Analyzing system performance and capacity using time-series data
  • Implementing SLIs, SLOs, and SLAs for service reliability
  • Debugging production issues using metrics and traces
  • Building custom exporters for application-specific metrics
  • Setting up federation for multi-cluster monitoring
  • Migrating from legacy monitoring to cloud-native solutions
  • Implementing cost monitoring and optimization tracking
  • Creating real-time operational dashboards for DevOps teams

Core Concepts

The Four Pillars of Observability

Modern observability is built on four fundamental pillars:

  1. Metrics: Numerical measurements of system behavior over time

    • Counter: Monotonically increasing values (requests served, errors)
    • Gauge: Point-in-time values that go up and down (memory usage, temperature)
    • Histogram: Distribution of values (request duration buckets)
    • Summary: Similar to histogram but calculates quantiles on client-side

  2. Logs: Discrete events with contextual information

    • Structured logging (JSON, key-value pairs)
    • Centralized log aggregation (ELK, Loki)
    • Correlation with metrics and traces

  3. Traces: Request flow through distributed systems

    • Span: Single unit of work with start/end time
    • Trace: Collection of spans representing end-to-end request
    • OpenTelemetry for distributed tracing

  4. Events: Significant occurrences in system lifecycle

    • Deployments, configuration changes
    • Scaling events, incidents
    • Business events and user actions

Prometheus Architecture

Prometheus is a pull-based monitoring system with key components:

Time-Series Database (TSDB)

  • Stores metrics as time-series data
  • Efficient compression and retention policies
  • Local storage with optional remote storage

Scrape Targets

  • Service discovery (Kubernetes, Consul, EC2, etc.)
  • Static configuration
  • Relabeling for flexible target selection

PromQL Query Engine

  • Powerful query language for metrics analysis
  • Aggregation, filtering, and mathematical operations
  • Range vectors and instant vectors

Alertmanager

  • Alert rule evaluation
  • Grouping, silencing, and routing
  • Integration with PagerDuty, Slack, email, webhooks

Exporters

  • Bridge between Prometheus and systems
  • Node exporter, cAdvisor, custom exporters
  • Third-party exporters for databases, services

Metric Labels and Cardinality

Labels are key-value pairs attached to metrics:

http_requests_total{method="GET", endpoint="/api/users", status="200"}

Label Best Practices:

  • Use labels for dimensions you query/aggregate by
  • Avoid high-cardinality labels (user IDs, timestamps)
  • Keep label names consistent across metrics
  • Use relabeling to normalize external labels

Cardinality Considerations:

  • Each unique label combination = new time-series
  • High cardinality = increased memory and storage
  • Monitor cardinality with prometheus_tsdb_symbol_table_size_bytes
  • Use recording rules to pre-aggregate high-cardinality metrics

Recording Rules

Pre-compute frequently-used or expensive queries:

groups:
  - name: api_performance
    interval: 30s
    rules:
      - record: api:http_request_duration_seconds:p99
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))
      - record: api:http_requests:rate5m
        expr: rate(http_requests_total[5m])

Benefits:

  • Faster dashboard loading
  • Reduced query load on Prometheus
  • Consistent metric naming conventions
  • Enable complex aggregations

Service Level Objectives (SLOs)

Define and track reliability targets:

SLI (Service Level Indicator): Metric measuring service quality

  • Availability: % of successful requests
  • Latency: % of requests under threshold
  • Throughput: Requests per second

SLO (Service Level Objective): Target for SLI

  • 99.9% availability (43.8 minutes downtime/month)
  • 95% of requests < 200ms
  • 1000 RPS sustained

SLA (Service Level Agreement): Contract with consequences

  • External commitments to customers
  • Financial penalties for SLO violations

Error Budget: Acceptable failure rate

  • Error budget = 100% - SLO
  • 99.9% SLO = 0.1% error budget
  • Use budget for innovation vs. reliability tradeoff

Prometheus Setup and Configuration

Basic Prometheus Configuration

# prometheus.yml
global:
  scrape_interval: 15s      # Default scrape interval
  evaluation_interval: 15s  # Alert rule evaluation interval
  external_labels:
    cluster: 'production'
    region: 'us-west-2'

# Alertmanager configuration
alerting:
  alertmanagers:
    - static_configs:
        - targets:
            - alertmanager:9093

# Load rules
rule_files:
  - 'rules/*.yml'
  - 'alerts/*.yml'

# Scrape configurations
scrape_configs:
  # Prometheus self-monitoring
  - job_name: 'prometheus'
    static_configs:
      - targets: ['localhost:9090']

  # Node exporter for system metrics
  - job_name: 'node'
    static_configs:
      - targets:
          - 'node1:9100'
          - 'node2:9100'
          - 'node3:9100'
    relabel_configs:
      - source_labels: [__address__]
        target_label: instance
        regex: '([^:]+):.*'
        replacement: '${1}'

  # Application metrics
  - job_name: 'api'
    static_configs:
      - targets: ['api-1:8080', 'api-2:8080', 'api-3:8080']
        labels:
          env: 'production'
          tier: 'backend'

Kubernetes Service Discovery

scrape_configs:
  # Kubernetes API server
  - job_name: 'kubernetes-apiservers'
    kubernetes_sd_configs:
      - role: endpoints
    scheme: https
    tls_config:
      ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
    bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
    relabel_configs:
      - source_labels: [__meta_kubernetes_namespace, __meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: default;kubernetes;https

  # Kubernetes pods with prometheus.io annotations
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Only scrape pods with prometheus.io/scrape: "true" annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      # Use the port from prometheus.io/port annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: (\d+)
        target_label: __address__
        replacement: ${1}:${2}
      # Use the path from prometheus.io/path annotation
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
      # Add namespace label
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      # Add pod name label
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

  # Kubernetes services
  - job_name: 'kubernetes-services'
    kubernetes_sd_configs:
      - role: service
    metrics_path: /probe
    params:
      module: [http_2xx]
    relabel_configs:
      - source_labels: [__meta_kubernetes_service_annotation_prometheus_io_probe]
        action: keep
        regex: true
      - source_labels: [__address__]
        target_label: __param_target
      - target_label: __address__
        replacement: blackbox-exporter:9115
      - source_labels: [__param_target]
        target_label: instance

Storage and Retention

# Storage configuration
storage:
  tsdb:
    path: /prometheus/data
    retention.time: 15d
    retention.size: 50GB

# Remote write for long-term storage
remote_write:
  - url: "https://prometheus-remote-storage.example.com/api/v1/write"
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/remote_storage_password
    queue_config:
      capacity: 10000
      max_shards: 50
      max_samples_per_send: 5000
    write_relabel_configs:
      # Drop high-cardinality metrics
      - source_labels: [__name__]
        regex: 'container_network_.*'
        action: drop

# Remote read for querying historical data
remote_read:
  - url: "https://prometheus-remote-storage.example.com/api/v1/read"
    basic_auth:
      username: prometheus
      password_file: /etc/prometheus/remote_storage_password
    read_recent: true

PromQL: The Prometheus Query Language

Instant Vectors and Selectors

# Basic metric selection
http_requests_total

# Filter by label
http_requests_total{job="api", status="200"}

# Regex matching
http_requests_total{status=~"2..|3.."}

# Negative matching
http_requests_total{status!="500"}

# Multiple label matchers
http_requests_total{job="api", method="GET", status=~"2.."}

Range Vectors and Aggregations

# 5-minute range vector
http_requests_total[5m]

# Rate of increase per second
rate(http_requests_total[5m])

# Increase over time window
increase(http_requests_total[1h])

# Average over time
avg_over_time(cpu_usage[5m])

# Max/Min over time
max_over_time(response_time_seconds[10m])
min_over_time(response_time_seconds[10m])

# Standard deviation
stddev_over_time(response_time_seconds[5m])

Aggregation Operators

# Sum across all instances
sum(rate(http_requests_total[5m]))

# Sum grouped by job
sum by (job) (rate(http_requests_total[5m]))

# Average grouped by multiple labels
avg by (job, instance) (cpu_usage)

# Count number of series
count(up == 1)

# Topk and bottomk
topk(5, rate(http_requests_total[5m]))
bottomk(3, node_memory_available_bytes)

# Quantile across instances
quantile(0.95, http_request_duration_seconds)

Mathematical Operations

# Arithmetic operations
(node_memory_total_bytes - node_memory_available_bytes) / node_memory_total_bytes * 100

# Comparison operators
http_request_duration_seconds > 0.5

# Logical operators
up == 1 and rate(http_requests_total[5m]) > 100

# Vector matching
rate(http_requests_total[5m]) / on(instance) group_left rate(http_responses_total[5m])

Advanced PromQL Patterns

# Request success rate
sum(rate(http_requests_total{status=~"2.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

# Error rate percentage
sum(rate(http_requests_total{status=~"5.."}[5m]))
/
sum(rate(http_requests_total[5m])) * 100

# Latency percentiles (histogram)
histogram_quantile(0.99,
  sum by (le) (rate(http_request_duration_seconds_bucket[5m]))
)

# Predict linear growth
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600)

# Detect anomalies with standard deviation
abs(cpu_usage - avg_over_time(cpu_usage[1h]))
>
3 * stddev_over_time(cpu_usage[1h])

# Calculate saturation (RED method)
sum(rate(cpu_seconds_total{mode!="idle"}[5m])) by (instance)
/
count(cpu_seconds_total{mode="idle"}) by (instance)

# Burn rate for SLO
(
  sum(rate(http_requests_total{status=~"5.."}[1h]))
  /
  sum(rate(http_requests_total[1h]))
)
>
(14.4 * (1 - 0.999))  # For 99.9% SLO

Alerting with Prometheus and Alertmanager

Alert Rule Definitions

# alerts/api_alerts.yml
groups:
  - name: api_alerts
    interval: 30s
    rules:
      # High error rate alert
      - alert: HighErrorRate
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (service)
          /
          sum(rate(http_requests_total[5m])) by (service)
          > 0.05
        for: 5m
        labels:
          severity: critical
          team: backend
        annotations:
          summary: "High error rate on {{ $labels.service }}"
          description: "Error rate is {{ $value | humanizePercentage }} on {{ $labels.service }}"
          runbook_url: "https://runbooks.example.com/HighErrorRate"

      # High latency alert
      - alert: HighLatency
        expr: |
          histogram_quantile(0.99,
            sum by (service, le) (rate(http_request_duration_seconds_bucket[5m]))
          ) > 1
        for: 10m
        labels:
          severity: warning
          team: backend
        annotations:
          summary: "High latency on {{ $labels.service }}"
          description: "P99 latency is {{ $value }}s on {{ $labels.service }}"

      # Service down alert
      - alert: ServiceDown
        expr: up{job="api"} == 0
        for: 2m
        labels:
          severity: critical
          team: sre
        annotations:
          summary: "Service {{ $labels.instance }} is down"
          description: "{{ $labels.job }} on {{ $labels.instance }} has been down for more than 2 minutes"

      # Disk space alert
      - alert: DiskSpaceLow
        expr: |
          (node_filesystem_avail_bytes{mountpoint="/"}
          /
          node_filesystem_size_bytes{mountpoint="/"}) * 100 < 10
        for: 5m
        labels:
          severity: warning
          team: sre
        annotations:
          summary: "Low disk space on {{ $labels.instance }}"
          description: "Disk space is {{ $value | humanize }}% on {{ $labels.instance }}"

      # Memory pressure alert
      - alert: HighMemoryUsage
        expr: |
          (1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100 > 90
        for: 10m
        labels:
          severity: warning
          team: sre
        annotations:
          summary: "High memory usage on {{ $labels.instance }}"
          description: "Memory usage is {{ $value | humanize }}% on {{ $labels.instance }}"

      # CPU saturation alert
      - alert: HighCPUUsage
        expr: |
          100 - (avg by (instance) (irate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 15m
        labels:
          severity: warning
          team: sre
        annotations:
          summary: "High CPU usage on {{ $labels.instance }}"
          description: "CPU usage is {{ $value | humanize }}% on {{ $labels.instance }}"

Alertmanager Configuration

# alertmanager.yml
global:
  resolve_timeout: 5m
  slack_api_url: 'https://hooks.slack.com/services/YOUR/SLACK/WEBHOOK'
  pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'

# Templates for notifications
templates:
  - '/etc/alertmanager/templates/*.tmpl'

# Route tree for alert distribution
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 12h
  receiver: 'team-default'

  routes:
    # Critical alerts go to PagerDuty
    - match:
        severity: critical
      receiver: 'pagerduty-critical'
      continue: true

    # Critical alerts also go to Slack
    - match:
        severity: critical
      receiver: 'slack-critical'
      group_wait: 0s

    # Warning alerts to Slack only
    - match:
        severity: warning
      receiver: 'slack-warnings'

    # Team-specific routing
    - match:
        team: backend
      receiver: 'team-backend'

    - match:
        team: frontend
      receiver: 'team-frontend'

    # Database alerts to DBA team
    - match_re:
        service: 'postgres|mysql|mongodb'
      receiver: 'team-dba'

# Alert receivers/integrations
receivers:
  - name: 'team-default'
    slack_configs:
      - channel: '#alerts'
        title: 'Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        send_resolved: true

  - name: 'pagerduty-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_SERVICE_KEY'
        description: '{{ .GroupLabels.alertname }}: {{ .GroupLabels.service }}'
        severity: '{{ .CommonLabels.severity }}'

  - name: 'slack-critical'
    slack_configs:
      - channel: '#incidents'
        title: 'CRITICAL: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.summary }}\n{{ .Annotations.description }}{{ end }}'
        color: 'danger'
        send_resolved: true

  - name: 'slack-warnings'
    slack_configs:
      - channel: '#monitoring'
        title: 'Warning: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
        color: 'warning'
        send_resolved: true

  - name: 'team-backend'
    slack_configs:
      - channel: '#team-backend'
        send_resolved: true
    email_configs:
      - to: 'backend-team@example.com'
        from: 'alertmanager@example.com'
        smarthost: 'smtp.gmail.com:587'
        auth_username: 'alertmanager@example.com'
        auth_password_file: '/etc/alertmanager/email_password'

  - name: 'team-frontend'
    slack_configs:
      - channel: '#team-frontend'
        send_resolved: true

  - name: 'team-dba'
    slack_configs:
      - channel: '#team-dba'
        send_resolved: true
    pagerduty_configs:
      - service_key: 'DBA_PAGERDUTY_KEY'

# Inhibition rules (suppress alerts)
inhibit_rules:
  # Inhibit warnings if critical alert is firing
  - source_match:
      severity: 'critical'
    target_match:
      severity: 'warning'
    equal: ['alertname', 'instance']

  # Don't alert on instance down if cluster is down
  - source_match:
      alertname: 'ClusterDown'
    target_match_re:
      alertname: 'InstanceDown|ServiceDown'
    equal: ['cluster']

Multi-Window Multi-Burn-Rate Alerts for SLOs

# SLO-based alerting using burn rate
groups:
  - name: slo_alerts
    interval: 30s
    rules:
      # Fast burn (1h window, 5m burn)
      - alert: ErrorBudgetBurnFast
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[1h]))
            /
            sum(rate(http_requests_total[1h]))
          ) > (14.4 * (1 - 0.999))
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          ) > (14.4 * (1 - 0.999))
        for: 2m
        labels:
          severity: critical
          slo: "99.9%"
        annotations:
          summary: "Fast error budget burn detected"
          description: "Error rate is burning through 99.9% SLO budget 14.4x faster than normal"

      # Slow burn (6h window, 30m burn)
      - alert: ErrorBudgetBurnSlow
        expr: |
          (
            sum(rate(http_requests_total{status=~"5.."}[6h]))
            /
            sum(rate(http_requests_total[6h]))
          ) > (6 * (1 - 0.999))
          and
          (
            sum(rate(http_requests_total{status=~"5.."}[30m]))
            /
            sum(rate(http_requests_total[30m]))
          ) > (6 * (1 - 0.999))
        for: 15m
        labels:
          severity: warning
          slo: "99.9%"
        annotations:
          summary: "Slow error budget burn detected"
          description: "Error rate is burning through 99.9% SLO budget 6x faster than normal"

Grafana Dashboards and Visualization

Dashboard JSON Structure

{
  "dashboard": {
    "title": "API Performance Dashboard",
    "tags": ["api", "performance", "production"],
    "timezone": "browser",
    "editable": true,
    "graphTooltip": 1,
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "timepicker": {
      "refresh_intervals": ["5s", "10s", "30s", "1m", "5m", "15m"],
      "time_options": ["5m", "15m", "1h", "6h", "12h", "24h", "7d"]
    },
    "templating": {
      "list": [
        {
          "name": "cluster",
          "type": "query",
          "datasource": "Prometheus",
          "query": "label_values(up, cluster)",
          "refresh": 1,
          "multi": false,
          "includeAll": false
        },
        {
          "name": "service",
          "type": "query",
          "datasource": "Prometheus",
          "query": "label_values(up{cluster=\"$cluster\"}, service)",
          "refresh": 1,
          "multi": true,
          "includeAll": true
        },
        {
          "name": "interval",
          "type": "interval",
          "query": "1m,5m,10m,30m,1h",
          "auto": true,
          "auto_count": 30,
          "auto_min": "10s"
        }
      ]
    },
    "panels": [
      {
        "id": 1,
        "title": "Request Rate",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 0, "y": 0},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=~\"$service\"}[$interval])) by (service)",
            "legendFormat": "{{ service }}",
            "refId": "A"
          }
        ],
        "yaxes": [
          {"format": "reqps", "label": "Requests/sec"},
          {"format": "short"}
        ],
        "legend": {
          "show": true,
          "values": true,
          "current": true,
          "avg": true,
          "max": true
        }
      },
      {
        "id": 2,
        "title": "Error Rate",
        "type": "graph",
        "gridPos": {"h": 8, "w": 12, "x": 12, "y": 0},
        "targets": [
          {
            "expr": "sum(rate(http_requests_total{service=~\"$service\",status=~\"5..\"}[$interval])) by (service) / sum(rate(http_requests_total{service=~\"$service\"}[$interval])) by (service) * 100",
            "legendFormat": "{{ service }} error %",
            "refId": "A"
          }
        ],
        "yaxes": [
          {"format": "percent", "label": "Error Rate"},
          {"format": "short"}
        ],
        "alert": {
          "conditions": [
            {
              "evaluator": {"params": [5], "type": "gt"},
              "operator": {"type": "and"},
              "query": {"params": ["A", "5m", "now"]},
              "reducer": {"params": [], "type": "avg"},
              "type": "query"
            }
          ],
          "executionErrorState": "alerting",
          "frequency": "1m",
          "handler": 1,
          "name": "High Error Rate",
          "noDataState": "no_data",
          "notifications": []
        }
      },
      {
        "id": 3,
        "title": "Latency Percentiles",
        "type": "graph",
        "gridPos": {"h": 8, "w": 24, "x": 0, "y": 8},
        "targets": [
          {
            "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (service, le))",
            "legendFormat": "{{ service }} p99",
            "refId": "A"
          },
          {
            "expr": "histogram_quantile(0.95, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (service, le))",
            "legendFormat": "{{ service }} p95",
            "refId": "B"
          },
          {
            "expr": "histogram_quantile(0.50, sum(rate(http_request_duration_seconds_bucket{service=~\"$service\"}[$interval])) by (service, le))",
            "legendFormat": "{{ service }} p50",
            "refId": "C"
          }
        ],
        "yaxes": [
          {"format": "s", "label": "Duration"},
          {"format": "short"}
        ]
      }
    ]
  }
}

RED Method Dashboard

The RED method focuses on Request rate, Error rate, and Duration:

{
  "panels": [
    {
      "title": "Request Rate (per service)",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total[$__rate_interval])) by (service)"
        }
      ]
    },
    {
      "title": "Error Rate % (per service)",
      "targets": [
        {
          "expr": "sum(rate(http_requests_total{status=~\"5..\"}[$__rate_interval])) by (service) / sum(rate(http_requests_total[$__rate_interval])) by (service) * 100"
        }
      ]
    },
    {
      "title": "Duration p99 (per service)",
      "targets": [
        {
          "expr": "histogram_quantile(0.99, sum(rate(http_request_duration_seconds_bucket[$__rate_interval])) by (service, le))"
        }
      ]
    }
  ]
}

USE Method Dashboard

The USE method monitors Utilization, Saturation, and Errors:

{
  "panels": [
    {
      "title": "CPU Utilization %",
      "targets": [
        {
          "expr": "100 - (avg by (instance) (irate(node_cpu_seconds_total{mode=\"idle\"}[$__rate_interval])) * 100)"
        }
      ]
    },
    {
      "title": "CPU Saturation (Load Average)",
      "targets": [
        {
          "expr": "node_load1"
        }
      ]
    },
    {
      "title": "Memory Utilization %",
      "targets": [
        {
          "expr": "(1 - (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes)) * 100"
        }
      ]
    },
    {
      "title": "Disk I/O Utilization %",
      "targets": [
        {
          "expr": "rate(node_disk_io_time_seconds_total[$__rate_interval]) * 100"
        }
      ]
    },
    {
      "title": "Network Errors",
      "targets": [
        {
          "expr": "rate(node_network_receive_errs_total[$__rate_interval]) + rate(node_network_transmit_errs_total[$__rate_interval])"
        }
      ]
    }
  ]
}

Exporters and Metric Collection

Node Exporter for System Metrics

# Install node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v1.6.1/node_exporter-1.6.1.linux-amd64.tar.gz
tar xvfz node_exporter-1.6.1.linux-amd64.tar.gz
cd node_exporter-1.6.1.linux-amd64
./node_exporter --web.listen-address=":9100" \
  --collector.filesystem.mount-points-exclude="^/(dev|proc|sys|var/lib/docker/.+)($|/)" \
  --collector.netclass.ignored-devices="^(veth.*|br.*|docker.*|lo)$"

Key Metrics from Node Exporter:

  • node_cpu_seconds_total: CPU usage by mode
  • node_memory_MemTotal_bytes, node_memory_MemAvailable_bytes: Memory
  • node_disk_io_time_seconds_total: Disk I/O
  • node_network_receive_bytes_total, node_network_transmit_bytes_total: Network
  • node_filesystem_size_bytes, node_filesystem_avail_bytes: Disk space

Custom Application Exporter (Python)

# app_exporter.py
from prometheus_client import start_http_server, Counter, Gauge, Histogram, Summary
import time
import random

# Define metrics
REQUEST_COUNT = Counter(
    'app_requests_total',
    'Total app requests',
    ['method', 'endpoint', 'status']
)

REQUEST_DURATION = Histogram(
    'app_request_duration_seconds',
    'Request duration in seconds',
    ['method', 'endpoint'],
    buckets=[0.01, 0.05, 0.1, 0.5, 1.0, 2.5, 5.0, 10.0]
)

ACTIVE_USERS = Gauge(
    'app_active_users',
    'Number of active users'
)

QUEUE_SIZE = Gauge(
    'app_queue_size',
    'Current queue size',
    ['queue_name']
)

DATABASE_CONNECTIONS = Gauge(
    'app_database_connections',
    'Number of database connections',
    ['pool', 'state']
)

CACHE_HITS = Counter(
    'app_cache_hits_total',
    'Total cache hits',
    ['cache_name']
)

CACHE_MISSES = Counter(
    'app_cache_misses_total',
    'Total cache misses',
    ['cache_name']
)

def simulate_metrics():
    """Simulate application metrics"""
    while True:
        # Simulate requests
        method = random.choice(['GET', 'POST', 'PUT', 'DELETE'])
        endpoint = random.choice(['/api/users', '/api/products', '/api/orders'])
        status = random.choice(['200', '200', '200', '400', '500'])

        REQUEST_COUNT.labels(method=method, endpoint=endpoint, status=status).inc()

        # Simulate request duration
        duration = random.uniform(0.01, 2.0)
        REQUEST_DURATION.labels(method=method, endpoint=endpoint).observe(duration)

        # Update gauges
        ACTIVE_USERS.set(random.randint(100, 1000))
        QUEUE_SIZE.labels(queue_name='jobs').set(random.randint(0, 50))
        QUEUE_SIZE.labels(queue_name='emails').set(random.randint(0, 20))

        # Database connection pool
        DATABASE_CONNECTIONS.labels(pool='main', state='active').set(random.randint(5, 20))
        DATABASE_CONNECTIONS.labels(pool='main', state='idle').set(random.randint(10, 30))

        # Cache metrics
        if random.random() > 0.3:
            CACHE_HITS.labels(cache_name='redis').inc()
        else:
            CACHE_MISSES.labels(cache_name='redis').inc()

        time.sleep(1)

if __name__ == '__main__':
    # Start metrics server on port 8000
    start_http_server(8000)
    print("Metrics server started on port 8000")
    simulate_metrics()

Custom Exporter (Go)

package main

import (
    "log"
    "math/rand"
    "net/http"
    "time"

    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promhttp"
)

var (
    requestsTotal = prometheus.NewCounterVec(
        prometheus.CounterOpts{
            Name: "app_requests_total",
            Help: "Total number of requests",
        },
        []string{"method", "endpoint", "status"},
    )

    requestDuration = prometheus.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "app_request_duration_seconds",
            Help:    "Request duration in seconds",
            Buckets: prometheus.ExponentialBuckets(0.01, 2, 10),
        },
        []string{"method", "endpoint"},
    )

    activeUsers = prometheus.NewGauge(
        prometheus.GaugeOpts{
            Name: "app_active_users",
            Help: "Number of active users",
        },
    )

    databaseConnections = prometheus.NewGaugeVec(
        prometheus.GaugeOpts{
            Name: "app_database_connections",
            Help: "Number of database connections",
        },
        []string{"pool", "state"},
    )
)

func init() {
    prometheus.MustRegister(requestsTotal)
    prometheus.MustRegister(requestDuration)
    prometheus.MustRegister(activeUsers)
    prometheus.MustRegister(databaseConnections)
}

func simulateMetrics() {
    ticker := time.NewTicker(1 * time.Second)
    defer ticker.Stop()

    for range ticker.C {
        // Simulate requests
        methods := []string{"GET", "POST", "PUT", "DELETE"}
        endpoints := []string{"/api/users", "/api/products", "/api/orders"}
        statuses := []string{"200", "200", "200", "400", "500"}

        method := methods[rand.Intn(len(methods))]
        endpoint := endpoints[rand.Intn(len(endpoints))]
        status := statuses[rand.Intn(len(statuses))]

        requestsTotal.WithLabelValues(method, endpoint, status).Inc()
        requestDuration.WithLabelValues(method, endpoint).Observe(rand.Float64() * 2)

        // Update gauges
        activeUsers.Set(float64(rand.Intn(900) + 100))
        databaseConnections.WithLabelValues("main", "active").Set(float64(rand.Intn(15) + 5))
        databaseConnections.WithLabelValues("main", "idle").Set(float64(rand.Intn(20) + 10))
    }
}

func main() {
    go simulateMetrics()

    http.Handle("/metrics", promhttp.Handler())
    log.Println("Starting metrics server on :8000")
    log.Fatal(http.ListenAndServe(":8000", nil))
}

PostgreSQL Exporter

# docker-compose.yml for postgres_exporter
version: '3.8'
services:
  postgres-exporter:
    image: prometheuscommunity/postgres-exporter
    environment:
      DATA_SOURCE_NAME: "postgresql://user:password@postgres:5432/dbname?sslmode=disable"
    ports:
      - "9187:9187"
    command:
      - '--collector.stat_statements'
      - '--collector.stat_database'
      - '--collector.replication'

Key PostgreSQL Metrics:

  • pg_up: Database reachability
  • pg_stat_database_tup_returned: Rows read
  • pg_stat_database_tup_inserted: Rows inserted
  • pg_stat_database_deadlocks: Deadlock count
  • pg_stat_replication_lag: Replication lag in seconds
  • pg_locks_count: Active locks

Blackbox Exporter for Probing

# blackbox.yml
modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
      valid_status_codes: [200]
      method: GET
      preferred_ip_protocol: "ip4"

  http_post_json:
    prober: http
    http:
      method: POST
      headers:
        Content-Type: application/json
      body: '{"key":"value"}'
      valid_status_codes: [200, 201]

  tcp_connect:
    prober: tcp
    timeout: 5s

  icmp:
    prober: icmp
    timeout: 5s
    icmp:
      preferred_ip_protocol: "ip4"

# Prometheus config for blackbox exporter
scrape_configs:
  - job_name: 'blackbox-http'
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets:
          - https://example.com
          - https://api.example.com/health
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox-exporter:9115

Best Practices

Metric Naming Conventions

Follow Prometheus naming best practices:

# Format: <namespace>_<subsystem>_<metric>_<unit>

# Good examples
http_requests_total              # Counter
http_request_duration_seconds    # Histogram
database_connections_active      # Gauge
cache_hits_total                 # Counter
memory_usage_bytes              # Gauge

# Include unit suffixes
_seconds, _bytes, _total, _ratio, _percentage

# Avoid
RequestCount                     # Use snake_case
http_requests                    # Missing _total for counter
request_time                     # Missing unit (should be _seconds)

Label Guidelines

# Good: Low cardinality labels
http_requests_total{method="GET", endpoint="/api/users", status="200"}

# Bad: High cardinality labels (avoid)
http_requests_total{user_id="12345", session_id="abc-def-ghi"}

# Good: Use bounded label values
http_requests_total{status_class="2xx"}

# Bad: Unbounded label values
http_requests_total{response_size="1234567"}

Recording Rule Patterns

groups:
  - name: performance_rules
    interval: 30s
    rules:
      # Pre-aggregate expensive queries
      - record: job:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (job)

      # Namespace aggregations
      - record: namespace:http_requests:rate5m
        expr: sum(rate(http_requests_total[5m])) by (namespace)

      # SLI calculations
      - record: job:http_requests_success:rate5m
        expr: sum(rate(http_requests_total{status=~"2.."}[5m])) by (job)

      - record: job:http_requests_error_rate:ratio
        expr: |
          sum(rate(http_requests_total{status=~"5.."}[5m])) by (job)
          /
          sum(rate(http_requests_total[5m])) by (job)

Alert Design Principles

  1. Alert on symptoms, not causes: Alert on user-facing issues
  2. Make alerts actionable: Include runbook links
  3. Use appropriate severity levels: Critical, warning, info
  4. Set proper thresholds: Based on historical data
  5. Include context in annotations: Help on-call engineers
  6. Group related alerts: Reduce alert fatigue
  7. Use inhibition rules: Suppress redundant alerts
  8. Test alert rules: Verify they fire when expected

Dashboard Best Practices

  1. One dashboard per audience: SRE, developers, business
  2. Use consistent time ranges: Make comparisons easier
  3. Include SLI/SLO metrics: Show business impact
  4. Add annotations for deploys: Correlate changes with metrics
  5. Use template variables: Make dashboards reusable
  6. Show trends and aggregates: Not just raw metrics
  7. Include links to runbooks: Enable quick response
  8. Use appropriate visualizations: Graphs, gauges, tables

High Availability Setup

# Prometheus HA with Thanos
# Deploy multiple Prometheus instances with same config
# Use Thanos to deduplicate and provide global view

# prometheus-1.yml
global:
  external_labels:
    cluster: 'prod'
    replica: '1'

# prometheus-2.yml
global:
  external_labels:
    cluster: 'prod'
    replica: '2'

# Thanos sidecar configuration
# Uploads blocks to object storage
# Provides StoreAPI for querying

Capacity Planning Queries

# Disk space exhaustion prediction
predict_linear(node_filesystem_free_bytes[1h], 4 * 3600) < 0

# Memory growth trend
predict_linear(node_memory_MemAvailable_bytes[1h], 24 * 3600)

# Request rate growth
predict_linear(sum(rate(http_requests_total[1h]))[24h:1h], 7 * 24 * 3600)

# Storage capacity planning
prometheus_tsdb_storage_blocks_bytes / (30 * 24 * 3600)

Advanced Patterns

Federation for Multi-Cluster Monitoring

# Global Prometheus federating from cluster Prometheus instances
scrape_configs:
  - job_name: 'federate'
    scrape_interval: 15s
    honor_labels: true
    metrics_path: '/federate'
    params:
      'match[]':
        - '{job="prometheus"}'
        - '{__name__=~"job:.*"}'  # All recording rules
    static_configs:
      - targets:
          - 'prometheus-us-west:9090'
          - 'prometheus-us-east:9090'
          - 'prometheus-eu-central:9090'

Cost Monitoring Pattern

# Track cloud costs with custom metrics
groups:
  - name: cost_tracking
    rules:
      - record: cloud:cost:hourly_rate
        expr: |
          (
            sum(kube_pod_container_resource_requests{resource="cpu"}) * 0.03  # CPU cost/hour
            +
            sum(kube_pod_container_resource_requests{resource="memory"} / 1024 / 1024 / 1024) * 0.005  # Memory cost/hour
          )

      - record: cloud:cost:monthly_estimate
        expr: cloud:cost:hourly_rate * 730  # Hours in average month

Custom SLO Implementation

# SLO: 99.9% availability for API
groups:
  - name: api_slo
    interval: 30s
    rules:
      # Success rate SLI
      - record: api:sli:success_rate
        expr: |
          sum(rate(http_requests_total{job="api",status=~"2.."}[5m]))
          /
          sum(rate(http_requests_total{job="api"}[5m]))

      # Error budget remaining (30 days)
      - record: api:error_budget:remaining
        expr: |
          1 - (
            (1 - api:sli:success_rate)
            /
            (1 - 0.999)
          )

      # Latency SLI (p99 < 500ms)
      - record: api:sli:latency_success_rate
        expr: |
          (
            histogram_quantile(0.99,
              sum(rate(http_request_duration_seconds_bucket{job="api"}[5m])) by (le)
            ) < 0.5
          )

Examples Summary

This skill includes 20+ comprehensive examples covering:

  1. Prometheus configuration (basic, Kubernetes SD, storage)
  2. PromQL queries (instant vectors, range vectors, aggregations)
  3. Mathematical operations and advanced patterns
  4. Alert rule definitions (error rate, latency, resource usage)
  5. Alertmanager configuration (routing, receivers, inhibition)
  6. Multi-window multi-burn-rate SLO alerts
  7. Grafana dashboard JSON (full dashboard, RED method, USE method)
  8. Custom exporters (Python, Go)
  9. Third-party exporters (PostgreSQL, Blackbox)
  10. Recording rules for performance
  11. Federation for multi-cluster monitoring
  12. Cost monitoring and SLO implementation
  13. High availability patterns
  14. Capacity planning queries

Skill Version: 1.0.0 Last Updated: October 2025 Skill Category: Observability, Monitoring, SRE, DevOps Compatible With: Prometheus, Grafana, Alertmanager, OpenTelemetry, Kubernetes

FAQ

What does the observability-monitoring skill do?

Comprehensive observability and monitoring skill covering Prometheus, Grafana, metrics collection, alerting, exporters, PromQL, and production monitoring patterns for distributed systems and cloud-native applications

How do I install observability-monitoring?

Run npx skills add https://github.com/manutej/luxor-claude-marketplace --skill observability-monitoring in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/observability-monitoring/ (personal) or .claude/skills/observability-monitoring/ (project).

Which AI agents can use observability-monitoring?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is observability-monitoring safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains observability-monitoring?

It is published by manutej in manutej/luxor-claude-marketplace. It has 455 recorded installs.

More from manutej/luxor-claude-marketplace

View all →

docker-compose-orchestration

manutej/luxor-claude-marketplace

Container orchestration with Docker Compose for multi-container applications, networking, volumes, and production deployment

DevOps & Cloud3K installs

postgresql-database-engineering

manutej/luxor-claude-marketplace

Comprehensive PostgreSQL database engineering skill covering indexing strategies, query optimization, performance tuning, partitioning, replication, backup and recovery, high availability, and production database management. Master advanced PostgreSQL features including MVCC, VACUUM operations, connection pooling, monitoring, and scalability patterns.

Data & Databases1.8K installs

playwright-visual-testing

manutej/luxor-claude-marketplace

Browser automation, visual testing, and screenshot validation using Playwright MCP server for accelerated web development. Master visual regression testing, automated UI testing, and cross-browser validation.

Testing & QA1.4K installs

golang-backend-development

manutej/luxor-claude-marketplace

Complete guide for Go backend development including concurrency patterns, web servers, database integration, microservices, and production deployment

Backend & APIs1.4K installs

jest-react-testing

manutej/luxor-claude-marketplace

Comprehensive React component testing with Jest and React Testing Library covering configuration, mocking strategies, async testing patterns, hooks testing, and integration testing best practices

Testing & QA1.2K installs

pytest-patterns

manutej/luxor-claude-marketplace

Python testing with pytest covering fixtures, parametrization, mocking, and test organization for reliable test suites

Testing & QA854 installs

Popular devops & cloud skills

View category →

microsoft-foundry

microsoft/azure-skills

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent create, managed harness agent create, GitHub Copilot harness agent, create agent, update agent, add tool to agent, invoke agent, agent insights, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure

OfficialDevOps & Cloud579.1K installs

azure-diagnostics

microsoft/azure-skills

Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.

OfficialDevOps & Cloud569.8K installs

azure-prepare

microsoft/azure-skills

Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.

OfficialDevOps & Cloud569K installs

azure-ai

microsoft/azure-skills

Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.

OfficialDevOps & Cloud568.8K installs

azure-deploy

microsoft/azure-skills

Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/deployment-plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.

OfficialDevOps & Cloud568.6K installs

azure-validate

microsoft/azure-skills

Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity permissions, and prerequisites before deploying. WHEN: validate my app, check deployment readiness, run preflight checks, verify configuration, check if ready to deploy, validate azure.yaml, validate Bicep, test before deploying, troubleshoot deployment errors, validate Azure Functions, validate function app, validate serverless deployment, verify RBAC roles, check role assignments, review managed identity permissions, what-if analysis, validate Container Apps deployment.

OfficialDevOps & Cloud568K installs