MCPInstall

runbookhermes-aiops-agent

by reason-machinesEducation & Science

Hermes-native AIOps agent for evidence-driven incident response, approval-gated remediation, and runbook learning

Installs
176
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/reason-machines/hermes-skills --skill runbookhermes-aiops-agent
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads runbookhermes-aiops-agent automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

RunbookHermes AIOps Agent Skill

Skill by ara.so — Hermes Skills collection.

RunbookHermes is a Hermes-native AIOps agent that specializes in incident response workflows. It extends Hermes Agent's runtime with evidence collection from observability tools (Prometheus, Loki, Jaeger), approval-gated remediation, checkpoint/rollback capabilities, and automatic runbook skill generation from resolved incidents.

What RunbookHermes Does

  • Evidence-driven incident analysis: Collects metrics, logs, traces, and deployment history
  • Approval-gated remediation: Requires human approval before risky actions
  • Runbook learning: Converts successful incident resolutions into reusable skills
  • Multi-channel intake: Accepts incidents from Web UI, Alertmanager, Feishu, WeCom, API
  • EvidenceStack context engine: Compresses observability data for model reasoning
  • IncidentMemory: Remembers service profiles, incident patterns, team preferences

Installation

Prerequisites

  • Python 3.10+
  • Hermes Agent (included as agent/ subdirectory)
  • Docker and Docker Compose (for local payment demo environment)

Clone and Install

git clone https://github.com/Tommy-yw/RunbookHermes.git
cd RunbookHermes

# Install dependencies
pip install -r requirements.txt

# Or use Poetry
poetry install

Environment Configuration

Create .env file in project root:

# Model provider (optional, for AI-assisted summaries)
OPENAI_API_KEY=${OPENAI_API_KEY}
OPENAI_BASE_URL=https://api.openai.com/v1
OPENAI_MODEL=gpt-4o

# Observability backends
PROMETHEUS_URL=http://localhost:9090
LOKI_URL=http://localhost:3100
JAEGER_URL=http://localhost:16686

# Deploy history backend
DEPLOY_BACKEND_TYPE=local_json
DEPLOY_HISTORY_PATH=./data/payment_demo/deploy_history.json

# Execution backend (for rollback/remediation)
EXECUTION_BACKEND_TYPE=local_reference
EXECUTION_CONFIG_PATH=./data/payment_demo/execution_config.json

# Feishu integration (optional)
FEISHU_APP_ID=${FEISHU_APP_ID}
FEISHU_APP_SECRET=${FEISHU_APP_SECRET}

# WeCom integration (optional)
WECOM_CORP_ID=${WECOM_CORP_ID}
WECOM_AGENT_SECRET=${WECOM_AGENT_SECRET}

# Web API
RUNBOOK_API_HOST=0.0.0.0
RUNBOOK_API_PORT=8000

Start Local Payment Demo Environment

cd demo/payment_system
docker-compose up -d
cd ../..

# Verify services are running
curl http://localhost:8001/health  # payment-service
curl http://localhost:8002/health  # coupon-service
curl http://localhost:8003/health  # order-service

Start RunbookHermes API Server

# From project root
python -m apps.runbook_api.main

# Or with uvicorn directly
uvicorn apps.runbook_api.main:app --host 0.0.0.0 --port 8000 --reload

Access Web Console at http://localhost:8000

Core Concepts

1. Hermes Profile Integration

RunbookHermes runs as a Hermes Agent profile located at profiles/runbook-hermes/:

# profiles/runbook-hermes/profile.yaml
name: runbook-hermes
version: 1.0.0
description: AIOps agent for incident response
persona: incident_responder
tools:
  - runbook-hermes
context_engine: evidence_stack
memory_provider: incident_memory

2. Evidence Collection Tools

The runbook-hermes tool plugin provides incident-response capabilities:

# Example: Query metrics evidence
from plugins.runbook_hermes.tools import query_metrics

evidence = query_metrics(
    service="payment-service",
    metric_type="http_5xx_rate",
    time_window="5m"
)

Available tools in the plugin:

  • query_metrics - Prometheus metrics collection
  • query_logs - Loki log search
  • query_traces - Jaeger trace analysis
  • get_deploy_history - Recent deployment records
  • create_checkpoint - Save system state before remediation
  • request_approval - Gate risky actions
  • execute_rollback - Controlled rollback execution
  • verify_recovery - Post-remediation health check

3. EvidenceStack Context Engine

Compresses observability data for model consumption:

from plugins.context_engine.evidence_stack.engine import EvidenceStackEngine

engine = EvidenceStackEngine()

# Add evidence
engine.add_evidence({
    "type": "metric",
    "service": "payment-service",
    "signal": "http_503_rate_spike",
    "value": "45 req/s",
    "severity": "critical"
})

# Get compressed context
context = engine.get_context()
# Returns: alert summary, key evidence, hypotheses, action plan

4. IncidentMemory Provider

Stores operational knowledge:

from plugins.memory.incident_memory.provider import IncidentMemoryProvider

memory = IncidentMemoryProvider()

# Remember service profile
memory.save_service_profile("payment-service", {
    "critical_metrics": ["http_5xx_rate", "p95_latency"],
    "dependencies": ["coupon-service", "order-service"],
    "rollback_safe": True
})

# Recall incident patterns
similar = memory.recall_similar_incidents(
    service="payment-service",
    symptom="http_503_spike"
)

Creating and Managing Incidents

Via Web Console

Navigate to http://localhost:8000/incidents/create and fill the form:

  • Service name
  • Severity (critical, high, medium, low)
  • Description
  • Alert data (optional)

Via API

import requests

response = requests.post("http://localhost:8000/api/incidents", json={
    "service": "payment-service",
    "severity": "critical",
    "description": "HTTP 503 rate spike detected",
    "alert": {
        "metric": "http_5xx_rate",
        "value": 45.2,
        "threshold": 5.0
    },
    "metadata": {
        "source": "alertmanager",
        "runbook_url": "https://wiki.example.com/payment-503"
    }
})

incident_id = response.json()["incident_id"]

Via Hermes CLI

# Run incident response through Hermes profile
hermes run \
  --profile runbook-hermes \
  --input "Payment service showing HTTP 503 errors at 45 req/s" \
  --context '{"service": "payment-service", "severity": "critical"}'

Via Alertmanager Webhook

Configure Alertmanager to send webhooks:

# alertmanager.yml
receivers:
  - name: runbook-hermes
    webhook_configs:
      - url: http://localhost:8000/gateway/alertmanager
        send_resolved: true

Approval Workflow

RunbookHermes gates risky actions behind approval:

# In your incident response logic
from runbook_hermes.approval import ApprovalManager

approval_mgr = ApprovalManager()

# Request approval for rollback
approval_id = approval_mgr.request_approval(
    incident_id="inc_001",
    action_type="rollback",
    target_service="payment-service",
    target_version="v1.2.3",
    risk_level="high",
    reason="Rollback to last known good version due to 503 spike",
    checkpoint_id="chk_001"
)

# Check approval status
status = approval_mgr.get_status(approval_id)
if status == "approved":
    # Execute rollback
    execute_rollback(service="payment-service", version="v1.2.3")

Approve via Web Console

Navigate to http://localhost:8000/approvals to review and approve/reject pending actions.

Approve via API

requests.post(f"http://localhost:8000/api/approvals/{approval_id}/approve", json={
    "operator": "alice",
    "comment": "Approved after verifying checkpoint"
})

Checkpoint and Rollback

Create Checkpoint Before Remediation

from runbook_hermes.checkpoint import CheckpointManager

checkpoint_mgr = CheckpointManager()

checkpoint = checkpoint_mgr.create(
    incident_id="inc_001",
    service="payment-service",
    snapshot_type="deployment",
    metadata={
        "current_version": "v1.2.4",
        "replica_count": 3,
        "config_hash": "abc123"
    }
)

Execute Rollback

from runbook_hermes.remediation import RemediationExecutor

executor = RemediationExecutor()

result = executor.rollback(
    service="payment-service",
    target_version="v1.2.3",
    checkpoint_id=checkpoint.id,
    dry_run=False
)

# Verify recovery
recovery_status = executor.verify_recovery(
    service="payment-service",
    expected_metrics={"http_5xx_rate": "<5"}
)

Runbook Skill Generation

After resolving an incident, generate a reusable skill:

from runbook_hermes.skills import SkillGenerator

generator = SkillGenerator()

skill = generator.generate_from_incident(
    incident_id="inc_001",
    skill_name="payment-http-503-rollback",
    trigger_conditions=["payment service 503 spike", "payment 5xx rate > 40"],
    steps=[
        "collect_evidence",
        "verify_deploy_change",
        "create_checkpoint",
        "request_approval",
        "rollback_deployment",
        "verify_recovery"
    ]
)

# Save to skills directory
skill.save("skills/runbooks/payment-http-503-rollback.yaml")

Generated skill format:

# skills/runbooks/payment-http-503-rollback.yaml
name: payment-http-503-rollback
version: 1.0.0
triggers:
  - payment service 503 spike
  - payment 5xx rate > 40
steps:
  - name: collect_evidence
    tool: query_metrics
    params:
      service: payment-service
      metric: http_5xx_rate
  - name: verify_deploy_change
    tool: get_deploy_history
    params:
      service: payment-service
      limit: 5
  - name: create_checkpoint
    tool: create_checkpoint
  - name: request_approval
    tool: request_approval
    risk_level: high
  - name: rollback_deployment
    tool: execute_rollback
  - name: verify_recovery
    tool: verify_recovery

Observability Integration

Prometheus Metrics

from integrations.observability.prometheus_adapter import PrometheusAdapter

prom = PrometheusAdapter(base_url="http://localhost:9090")

# Query current 5xx rate
result = prom.query_range(
    query='rate(http_requests_total{status=~"5..", service="payment-service"}[5m])',
    start="-15m",
    end="now",
    step="30s"
)

# Extract evidence
if result.has_spike(threshold=5.0):
    evidence = {
        "type": "metric",
        "signal": "http_5xx_spike",
        "max_value": result.max_value(),
        "timestamp": result.max_timestamp()
    }

Loki Logs

from integrations.observability.loki_adapter import LokiAdapter

loki = LokiAdapter(base_url="http://localhost:3100")

# Search error logs
logs = loki.query_range(
    query='{service="payment-service"} |= "error" | json',
    start="-15m",
    limit=100
)

# Extract patterns
error_patterns = logs.extract_patterns(min_frequency=5)

Jaeger Traces

from integrations.observability.jaeger_adapter import JaegerAdapter

jaeger = JaegerAdapter(base_url="http://localhost:16686")

# Find slow traces
traces = jaeger.search_traces(
    service="payment-service",
    start="-15m",
    min_duration="500ms",
    limit=20
)

# Analyze error traces
for trace in traces.with_errors():
    root_cause_span = trace.find_slowest_span()

Running Hermes Agent with RunbookHermes Profile

Direct CLI Invocation

# Run incident triage
hermes run \
  --profile runbook-hermes \
  --input "Payment service p95 latency is 2.5s, normal is 200ms" \
  --verbose

# Run with specific tool selection
hermes run \
  --profile runbook-hermes \
  --input "Check payment service deployment history" \
  --tools query_metrics,get_deploy_history

Programmatic Invocation

from agent.runtime import HermesRuntime
from agent.config import AgentConfig

config = AgentConfig(
    profile="runbook-hermes",
    tools=["runbook-hermes"],
    context_engine="evidence_stack",
    memory_provider="incident_memory"
)

runtime = HermesRuntime(config)

response = runtime.run(
    input_text="Investigate payment-service HTTP 503 spike",
    context={
        "service": "payment-service",
        "incident_id": "inc_001",
        "severity": "critical"
    }
)

print(response.final_answer)
print(response.evidence_chain)
print(response.recommended_actions)

Common Patterns

Pattern 1: Full Incident Response Workflow

from runbook_hermes.workflow import IncidentResponseWorkflow

workflow = IncidentResponseWorkflow()

# Execute end-to-end
result = workflow.execute(
    service="payment-service",
    symptom="http_503_spike",
    severity="critical",
    auto_approve=False  # Require human approval
)

print(f"Root cause: {result.root_cause}")
print(f"Remediation: {result.remediation_action}")
print(f"Status: {result.status}")

Pattern 2: Evidence-Driven Diagnosis

from runbook_hermes.diagnosis import EvidenceDiagnosis

diagnosis = EvidenceDiagnosis(service="payment-service")

# Collect all evidence types
diagnosis.collect_metrics(time_window="15m")
diagnosis.collect_logs(time_window="15m", error_only=True)
diagnosis.collect_traces(time_window="15m", min_duration="500ms")
diagnosis.collect_deploy_history(limit=10)

# Analyze
root_cause = diagnosis.analyze()

print(f"Most likely cause: {root_cause.hypothesis}")
print(f"Confidence: {root_cause.confidence}")
print(f"Supporting evidence: {root_cause.evidence_ids}")

Pattern 3: Safe Remediation with Approval

from runbook_hermes.remediation import SafeRemediation

remediation = SafeRemediation(incident_id="inc_001")

# Plan action
plan = remediation.plan_rollback(
    service="payment-service",
    target_version="v1.2.3"
)

# Create checkpoint
checkpoint = remediation.create_checkpoint()

# Request approval (blocks until human decision)
approval = remediation.request_approval(
    action=plan,
    checkpoint=checkpoint,
    timeout_minutes=30
)

if approval.is_approved():
    # Execute with dry-run first
    dry_run_result = remediation.execute(dry_run=True)
    
    if dry_run_result.success:
        # Real execution
        result = remediation.execute(dry_run=False)
        
        # Verify recovery
        if remediation.verify_recovery():
            print("Remediation successful")
        else:
            # Auto-rollback to checkpoint
            remediation.restore_checkpoint(checkpoint.id)

Pattern 4: Multi-Service Impact Analysis

from runbook_hermes.topology import ServiceTopology

topology = ServiceTopology()

# Build dependency graph
graph = topology.build_graph(
    root_service="payment-service",
    depth=2
)

# Analyze impact
impact = topology.analyze_impact(
    failing_service="payment-service",
    failure_type="http_503"
)

print(f"Directly impacted: {impact.direct}")
print(f"Indirectly impacted: {impact.indirect}")
print(f"Suggested investigation order: {impact.priority_list}")

Configuration Reference

RunbookHermes Config File

Create config/runbook_hermes.yaml:

# Incident response settings
incident:
  auto_create_from_alert: true
  default_severity: high
  evidence_collection_timeout: 300  # seconds
  
# Evidence collection
evidence:
  metrics:
    enabled: true
    time_window: 15m
    retention_days: 30
  logs:
    enabled: true
    max_lines: 1000
    error_patterns_only: false
  traces:
    enabled: true
    sample_limit: 100
    min_duration: 200ms
    
# Approval settings
approval:
  required_for:
    - rollback
    - restart
    - config_change
    - scale_down
  auto_approve_on_critical: false
  approval_timeout_minutes: 30
  require_checkpoint: true
  
# Remediation
remediation:
  dry_run_first: true
  verify_recovery: true
  recovery_check_interval: 30  # seconds
  max_recovery_wait: 300  # seconds
  auto_rollback_on_failure: true
  
# Runbook skill generation
skills:
  auto_generate: true
  min_success_count: 1
  output_dir: skills/runbooks
  
# Model-assisted analysis (optional)
model:
  enabled: true
  provider: openai
  temperature: 0.3
  max_tokens: 2000

Tool Configuration

# plugins/runbook_hermes/config.yaml
tools:
  query_metrics:
    timeout: 30
    max_results: 1000
  query_logs:
    timeout: 60
    max_lines: 5000
  query_traces:
    timeout: 45
    max_traces: 200
  execute_rollback:
    require_approval: true
    require_checkpoint: true
    dry_run_first: true

Troubleshooting

Issue: Evidence collection returns empty results

Cause: Observability backends not reachable or no data in time window

Solution:

# Test backend connectivity
from integrations.observability.health import check_backends

health = check_backends()
print(f"Prometheus: {health['prometheus']}")
print(f"Loki: {health['loki']}")
print(f"Jaeger: {health['jaeger']}")

# Verify time window
# Ensure time_window matches your metric retention
evidence = query_metrics(
    service="payment-service",
    time_window="1h"  # Increase window
)

Issue: Approval requests timeout

Cause: No operator reviewing approvals in time

Solution:

# config/runbook_hermes.yaml
approval:
  approval_timeout_minutes: 60  # Increase timeout
  fallback_to_auto_reject: false  # Prevent auto-reject
  
# Or configure notification
notification:
  on_approval_request:
    - type: feishu
      webhook_url: ${FEISHU_APPROVAL_WEBHOOK}

Issue: Runbook skills not generating

Cause: Incident not marked as resolved or missing evidence

Solution:

# Explicitly mark incident resolved
from runbook_hermes.incident import IncidentManager

mgr = IncidentManager()
mgr.mark_resolved(
    incident_id="inc_001",
    resolution="Rolled back to v1.2.3",
    root_cause="Bad deployment v1.2.4"
)

# Manually trigger skill generation
from runbook_hermes.skills import SkillGenerator

generator = SkillGenerator()
skill = generator.generate_from_incident("inc_001")
skill.save()

Issue: Model-assisted summaries failing

Cause: Model API key not configured or endpoint unreachable

Solution:

# Verify environment variables
echo $OPENAI_API_KEY
echo $OPENAI_BASE_URL

# Test model connectivity
curl $OPENAI_BASE_URL/models \
  -H "Authorization: Bearer $OPENAI_API_KEY"

# Disable model if not needed
# config/runbook_hermes.yaml
model:
  enabled: false  # Fall back to evidence-only mode

Issue: Hermes profile not found

Cause: Profile directory not in Hermes search path

Solution:

# Add RunbookHermes profiles to Hermes config
export HERMES_PROFILE_PATH="./profiles/runbook-hermes:$HERMES_PROFILE_PATH"

# Or copy profile to Hermes profiles directory
cp -r profiles/runbook-hermes ~/.hermes/profiles/

Debug Mode

Enable verbose logging:

import logging
logging.basicConfig(level=logging.DEBUG)

# Or set environment variable
export RUNBOOK_HERMES_LOG_LEVEL=DEBUG

# Run with debug output
hermes run \
  --profile runbook-hermes \
  --input "Debug payment service issue" \
  --debug \
  --trace-tools

Advanced Usage

Custom Tool Integration

Add domain-specific tools:

# plugins/runbook_hermes/custom_tools.py
from agent.tools import Tool, ToolParameter

class CheckDatabaseConnectionTool(Tool):
    name = "check_database_connection"
    description = "Verify database connectivity and connection pool status"
    
    parameters = [
        ToolParameter(name="service", type="string", required=True),
        ToolParameter(name="db_name", type="string", required=True)
    ]
    
    def execute(self, service: str, db_name: str) -> dict:
        # Your custom logic
        return {
            "status": "healthy",
            "active_connections": 25,
            "max_connections": 100
        }

# Register tool
from plugins.runbook_hermes.registry import register_tool
register_tool(CheckDatabaseConnectionTool())

Custom Evidence Type

# runbook_hermes/evidence/custom_evidence.py
from runbook_hermes.evidence import EvidenceCollector

class CostEvidenceCollector(EvidenceCollector):
    def collect(self, service: str, time_window: str) -> dict:
        # Collect cost metrics from billing API
        return {
            "type": "cost_spike",
            "service": service,
            "cost_increase_pct": 150,
            "period": time_window
        }

# Register collector
from runbook_hermes.evidence import register_collector
register_collector("cost", CostEvidenceCollector())

This skill enables AI coding agents to help developers deploy, configure, and operate RunbookHermes for production incident response with Hermes Agent integration.

FAQ

What does the runbookhermes-aiops-agent skill do?

Hermes-native AIOps agent for evidence-driven incident response, approval-gated remediation, and runbook learning

How do I install runbookhermes-aiops-agent?

Run npx skills add https://github.com/reason-machines/hermes-skills --skill runbookhermes-aiops-agent in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/runbookhermes-aiops-agent/ (personal) or .claude/skills/runbookhermes-aiops-agent/ (project).

Which AI agents can use runbookhermes-aiops-agent?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is runbookhermes-aiops-agent safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains runbookhermes-aiops-agent?

It is published by reason-machines in reason-machines/hermes-skills. It has 176 recorded installs.

More from reason-machines/hermes-skills

View all →

hermes-agent-self-evolution

reason-machines/hermes-skills

Evolutionary self-improvement for Hermes Agent using DSPy + GEPA to optimize skills, prompts, and code

AI & Agents323 installs

hermes-webui-agent

reason-machines/hermes-skills

Expert in deploying, configuring, and using Hermes WebUI—a web interface for Hermes Agent with persistent memory, scheduled jobs, and multi-platform messaging.

267 installs

awesome-hermes-agent-ecosystem

reason-machines/hermes-skills

Expert knowledge of the Hermes Agent ecosystem, skills, plugins, tools, integrations, and deployment patterns for building and extending AI agents

AI & Agents239 installs

hermes-agent-framework

reason-machines/hermes-skills

Hermes Agent framework by Nous Research - self-improving AI agent with built-in learning loop, three-layer memory, and automatic skill evolution

AI & Agents222 installs

hermes-workspace-ai-agent-ui

reason-machines/hermes-skills

Native web workspace for Hermes Agent with chat, terminal, memory, skills, inspector, and multi-agent orchestration

AI & Agents218 installs

hermes-web-ui-dashboard

reason-machines/hermes-skills

Web dashboard for managing Hermes Agent multi-platform AI chat sessions, analytics, scheduled jobs, and platform channels

Data & Databases210 installs

Popular education & science skills

View category →

teach

mattpocock/skills

Teach the user a new skill or concept, within this workspace.

Education & Science874K installs

scaffold-exercises

mattpocock/skills

Create exercise directory structures with sections, problems, solutions, and explainers that pass linting. Use when user wants to scaffold exercises, create exercise stubs, or set up a new course section.

Education & Science488.2K installs

paper-context-resolver

lllllllama/rigorpilot-skills

Rigor Paper Context helper for README-first deep learning repo reproduction. Use only when the README and repository files leave a narrow reproduction-critical gap and the task is to resolve a specific paper detail such as dataset split, preprocessing, evaluation protocol, checkpoint mapping, or runtime assumption from primary paper sources while recording conflicts. Do not use for general paper summary, repo scanning, environment setup, command execution, title-only paper lookup, or replacing README guidance by default.

Education & Science478.3K installs

caveman-learn

juliusbrussee/caveman

Act on a Caveman learn report - review the ranked token sinks, apply cost-lowering fixes with per-edit consent, and report what those fixes returned. Use when asked to lower an agent's token cost, what caveman has saved, to trim a heavy CLAUDE.md, or to offload re-pasted context into cavemem.

Education & Science107.9K installs

gsap-plugins

greensock/gsap-skills

Official GSAP skill for GSAP plugins — registration, ScrollToPlugin, ScrollSmoother, Flip, Draggable, Inertia, Observer, SplitText, ScrambleText, SVG and physics plugins, CustomEase, EasePack, CustomWiggle, CustomBounce, GSDevTools. Use when the user asks about a GSAP plugin, scroll-to, flip animations, draggable, SVG drawing, or plugin registration.

Education & Science67.9K installs

teach-impeccable

pbakaus/impeccable

DEPRECATED: Use /impeccable teach instead. This command has been folded into the impeccable skill.

Education & Science50K installs