MCPInstall

monitoring-observability

Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.

Installs
262
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/yonatangross/orchestkit --skill monitoring-observability
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads monitoring-observability automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Monitoring & Observability

A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the vendor for everything else. Start at references/ork-delta.md.

Upstream coverage (do not restate)

These topics are fully covered first-party. Read the source, do not add a local copy.

TopicFirst-party source
Prometheus metric types, RED method, cardinality, PromQLhttps://prometheus.io/docs/practices/
Alertmanager grouping, inhibition, escalation, runbookshttps://prometheus.io/docs/alerting/latest/configuration/
Grafana dashboards, Loki and LogQL, Promtailhttps://grafana.com/docs/
OpenTelemetry spans, sampling, context propagationhttps://opentelemetry.io/docs/
Langfuse Python SDK (@observe, as_type, score_current_span, should_export_span, LangfuseMedia)https://langfuse.com/docs/sdk/python
Langfuse v2 to v4 Python and v3 to v5 JS migration pathshttps://langfuse.com/docs/sdk/python/v4-migration
Langfuse self-hosting (ClickHouse, Redis, S3, Helm)https://langfuse.com/docs/deployment/self-host
Langfuse cost tracking, model pricing, Metrics API v2https://langfuse.com/docs/model-usage-and-cost
Langfuse scores, online evaluators, annotation queues, prompt managementhttps://langfuse.com/docs/scores/overview
Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit)https://langfuse.com/docs/integrations
Agent Graphs, observation types, rendered tool callshttps://langfuse.com/docs/tracing-features/agent-graphs
PSI, KS test, KL and JS divergence, Wasserstein, embedding drifthttps://www.evidentlyai.com/blog/data-drift-detection-large-datasets
EWMA control chartshttps://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm
structlog, Winston, correlation IDs, log samplinghttps://www.structlog.org/en/stable/

Quick Reference

CategoryRulesImpactWhen to Use
Infrastructure Monitoring1CRITICALGrafana dashboards, Golden Signals, SLO/SLI
LLM Observability1HIGHLangfuse tracing, observation types, agent graphs
Silent Failures3HIGHTool skipping, quality degradation, loop/token spike alerting

Total: 5 rules across 3 categories. Drift detection, cost tracking, eval scoring, Prometheus instrumentation and alert-rule authoring moved to the upstream sources listed above.

Quick Start

# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
from langfuse import observe, get_client

@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
    get_client().update_current_trace(
        user_id="user_123", session_id="session_abc",
        tags=["production", "orchestkit"],
    )
    result = await llm.generate(content)
    get_client().score_current_span(name="response_quality", value=0.85)
    return result
# Prometheus RED method, wired the way this repo expects (bounded labels only)
from prometheus_client import Counter, Histogram

http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
    buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])

Infrastructure Monitoring

Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.

RuleFileKey Pattern
Grafana Dashboardsrules/monitoring-grafana.mdGolden Signals, SLO/SLI, health checks

CC 2.1.161 — OTEL resource attributes as metric labels: OTEL_RESOURCE_ATTRIBUTES values are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.

LLM Observability

Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are upstream; what stays here is how this repo wires traces.

RuleFileKey Pattern
Langfuse Tracesrules/llm-langfuse-traces.md@observe decorator, OTEL spans, agent graphs

Silent Failures

Detection and alerting for silent failures in LLM agents.

RuleFileKey Pattern
Tool Skippingrules/silent-tool-skipping.mdExpected vs actual tool calls, Langfuse traces
Quality Degradationrules/silent-degraded-quality.mdHeuristics + LLM-as-judge, z-score baselines
Silent Alertingrules/silent-alerting.mdLoop detection, token spikes, escalation workflow

CC 2.1.169 — OTEL client-cert paths require trust: untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project .claude/settings.json, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.

Key Decisions

DecisionRecommendationRationale
Metric methodologyRED method (Rate, Errors, Duration)Industry standard, covers essential service health
Log formatStructured JSONMachine-parseable, supports log aggregation
TracingOpenTelemetryVendor-neutral, auto-instrumentation, broad ecosystem
LLM observabilityLangfuse (not LangSmith)Open-source, self-hosted, built-in prompt management
LLM tracing API@observe(as_type=...) + score_current_span()v4: semantic types, inline scoring, span filtering
Langfuse APIsObservations API v2 + Metrics API v2v4 (Mar 2026): faster querying, aggregations at scale
Hook telemetry transportJSONL under ~/.claude/analytics/, never an SDK in-processHooks are per-event processes; SDK init would be paid on every spawn (references/ork-delta.md)

Detailed Documentation

ResourceDescription
references/ork-delta.mdStart here. Floors, house decisions and scars that upstream docs do not carry
references/langfuse-js-v5.mdJS/TS SDK v5 delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code
references/experiments-api.mdLangfuse experiments and dataset runs as this repo uses them
references/evaluation-scores.mdScore shapes and scoring pipeline wiring
references/session-tracking.mdSession and user grouping across multi-step workflows
references/metrics-collection.mdClaude Code OTEL metric inventory and collector-side joins
references/dashboards.mdDashboard layout conventions
references/structured-logging.mdStructured log field conventions
references/dev-agent-lens.mdLiteLLM proxy layer for API-boundary observability
examples/orchestkit-monitoring-dashboard.mdWorked monitoring dashboard example
scripts/Templates: Prometheus, OpenTelemetry, health checks, Langfuse

Related Skills

  • defense-in-depth - Layer 8 observability as part of security architecture
  • devops-deployment - Observability integration with CI/CD and Kubernetes
  • resilience-patterns - Monitoring circuit breakers and failure scenarios
  • llm-evaluation - Evaluation patterns that integrate with Langfuse scoring
  • caching - Caching strategies that reduce costs tracked by Langfuse

FAQ

What does the monitoring-observability skill do?

Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.

How do I install monitoring-observability?

Run npx skills add https://github.com/yonatangross/orchestkit --skill monitoring-observability in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/monitoring-observability/ (personal) or .claude/skills/monitoring-observability/ (project).

Which AI agents can use monitoring-observability?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is monitoring-observability safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains monitoring-observability?

It is published by yonatangross in yonatangross/orchestkit. It has 262 recorded installs.

More from yonatangross/orchestkit

View all →

devops-deployment

yonatangross/orchestkit

Use when setting up CI/CD pipelines, containerizing applications, deploying to Kubernetes, or writing infrastructure as code. DevOps & Deployment covers GitHub Actions, Docker, Helm, and Terraform patterns.

DevOps & Cloud1.6K installs

responsive-patterns

yonatangross/orchestkit

Responsive design with Container Queries, fluid typography, cqi/cqb units, subgrid, intrinsic layouts, foldable devices, and mobile-first patterns for React applications. Use when building responsive layouts or container queries.

Frontend & Design1.6K installs

architecture-decision-record

yonatangross/orchestkit

ADR templates in the Nygard format with context, decision, consequences, and alternatives. Use when writing ADRs, recording an architectural decision, or evaluating options.

ui-components

yonatangross/orchestkit

UI component library patterns for shadcn/ui and Radix Primitives. Use when building accessible component libraries, customizing shadcn components, using Radix unstyled primitives, or creating design system foundations.

Frontend & Design1.5K installs

rag-retrieval

yonatangross/orchestkit

Retrieval-Augmented Generation patterns for grounded LLM responses. Use when building RAG pipelines, embedding documents, implementing hybrid search, contextual retrieval, HyDE, agentic RAG, multimodal RAG, query decomposition, reranking, or pgvector search.

AI & Agents1.5K installs

zustand-patterns

yonatangross/orchestkit

Reference for Zustand 5.x state management including slices, middleware, Immer, useShallow, persistence, selectors, and devtools integration. Documents 7 core patterns with TypeScript examples and anti-patterns. Use when building React state management with Zustand instead of Redux.

Frontend & Design1.4K installs

Popular devops & cloud skills

View category →

microsoft-foundry

microsoft/azure-skills

Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent create, managed harness agent create, GitHub Copilot harness agent, create agent, update agent, add tool to agent, invoke agent, agent insights, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure

OfficialDevOps & Cloud579.1K installs

azure-diagnostics

microsoft/azure-skills

Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.

OfficialDevOps & Cloud569.8K installs

azure-prepare

microsoft/azure-skills

Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.

OfficialDevOps & Cloud569K installs

azure-ai

microsoft/azure-skills

Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.

OfficialDevOps & Cloud568.8K installs

azure-deploy

microsoft/azure-skills

Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/deployment-plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.

OfficialDevOps & Cloud568.6K installs

azure-validate

microsoft/azure-skills

Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity permissions, and prerequisites before deploying. WHEN: validate my app, check deployment readiness, run preflight checks, verify configuration, check if ready to deploy, validate azure.yaml, validate Bicep, test before deploying, troubleshoot deployment errors, validate Azure Functions, validate function app, validate serverless deployment, verify RBAC roles, check role assignments, review managed identity permissions, what-if analysis, validate Container Apps deployment.

OfficialDevOps & Cloud568K installs