
monitoring-observability
Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
- Installs
- 262
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/yonatangross/orchestkit --skill monitoring-observability- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads monitoring-observability automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Monitoring & Observability
A wrap around Prometheus, Grafana, OpenTelemetry and Langfuse, not a re-teaching of them. This
skill carries OrchestKit's delta (version floors, house decisions, scars) and points at the
vendor for everything else. Start at references/ork-delta.md.
Upstream coverage (do not restate)
These topics are fully covered first-party. Read the source, do not add a local copy.
| Topic | First-party source |
|---|---|
| Prometheus metric types, RED method, cardinality, PromQL | https://prometheus.io/docs/practices/ |
| Alertmanager grouping, inhibition, escalation, runbooks | https://prometheus.io/docs/alerting/latest/configuration/ |
| Grafana dashboards, Loki and LogQL, Promtail | https://grafana.com/docs/ |
| OpenTelemetry spans, sampling, context propagation | https://opentelemetry.io/docs/ |
Langfuse Python SDK (@observe, as_type, score_current_span, should_export_span, LangfuseMedia) | https://langfuse.com/docs/sdk/python |
| Langfuse v2 to v4 Python and v3 to v5 JS migration paths | https://langfuse.com/docs/sdk/python/v4-migration |
| Langfuse self-hosting (ClickHouse, Redis, S3, Helm) | https://langfuse.com/docs/deployment/self-host |
| Langfuse cost tracking, model pricing, Metrics API v2 | https://langfuse.com/docs/model-usage-and-cost |
| Langfuse scores, online evaluators, annotation queues, prompt management | https://langfuse.com/docs/scores/overview |
| Langfuse framework integrations (LangChain, LangGraph, CrewAI, Pydantic AI, Bedrock, LiveKit) | https://langfuse.com/docs/integrations |
| Agent Graphs, observation types, rendered tool calls | https://langfuse.com/docs/tracing-features/agent-graphs |
| PSI, KS test, KL and JS divergence, Wasserstein, embedding drift | https://www.evidentlyai.com/blog/data-drift-detection-large-datasets |
| EWMA control charts | https://www.itl.nist.gov/div898/handbook/pmc/section3/pmc324.htm |
| structlog, Winston, correlation IDs, log sampling | https://www.structlog.org/en/stable/ |
Quick Reference
| Category | Rules | Impact | When to Use |
|---|---|---|---|
| Infrastructure Monitoring | 1 | CRITICAL | Grafana dashboards, Golden Signals, SLO/SLI |
| LLM Observability | 1 | HIGH | Langfuse tracing, observation types, agent graphs |
| Silent Failures | 3 | HIGH | Tool skipping, quality degradation, loop/token spike alerting |
Total: 5 rules across 3 categories. Drift detection, cost tracking, eval scoring, Prometheus instrumentation and alert-rule authoring moved to the upstream sources listed above.
Quick Start
# Langfuse v4 LLM tracing: semantic as_type plus inline scoring
from langfuse import observe, get_client
@observe(as_type="generation", name="analyze_content")
async def analyze_content(content: str):
get_client().update_current_trace(
user_id="user_123", session_id="session_abc",
tags=["production", "orchestkit"],
)
result = await llm.generate(content)
get_client().score_current_span(name="response_quality", value=0.85)
return result
# Prometheus RED method, wired the way this repo expects (bounded labels only)
from prometheus_client import Counter, Histogram
http_requests = Counter('http_requests_total', 'Total requests', ['method', 'endpoint', 'status'])
http_duration = Histogram('http_request_duration_seconds', 'Request latency',
buckets=[0.01, 0.05, 0.1, 0.5, 1, 2, 5])
Infrastructure Monitoring
Dashboard and health-check patterns. Metric instrumentation and alert-rule syntax are upstream.
| Rule | File | Key Pattern |
|---|---|---|
| Grafana Dashboards | rules/monitoring-grafana.md | Golden Signals, SLO/SLI, health checks |
CC 2.1.161 — OTEL resource attributes as metric labels:
OTEL_RESOURCE_ATTRIBUTESvalues are now attached as labels on metric datapoints, so usage metrics can be sliced by custom dimensions (team, repo, environment). Add label selectors to dashboards for multi-tenant / per-team cost and usage tracking.
LLM Observability
Langfuse-based tracing for LLM applications. Cost tracking, scoring and drift statistics are upstream; what stays here is how this repo wires traces.
| Rule | File | Key Pattern |
|---|---|---|
| Langfuse Traces | rules/llm-langfuse-traces.md | @observe decorator, OTEL spans, agent graphs |
Silent Failures
Detection and alerting for silent failures in LLM agents.
| Rule | File | Key Pattern |
|---|---|---|
| Tool Skipping | rules/silent-tool-skipping.md | Expected vs actual tool calls, Langfuse traces |
| Quality Degradation | rules/silent-degraded-quality.md | Heuristics + LLM-as-judge, z-score baselines |
| Silent Alerting | rules/silent-alerting.md | Loop detection, token spikes, escalation workflow |
CC 2.1.169 — OTEL client-cert paths require trust: untrusted project settings can no longer set OTEL client-certificate paths without a trust confirmation. If your OTEL exporter uses client certs configured in project
.claude/settings.json, expect a one-time trust prompt on first use in an untrusted project — telemetry silently not flowing after 2.1.169 is usually this gate, not the collector.
Key Decisions
| Decision | Recommendation | Rationale |
|---|---|---|
| Metric methodology | RED method (Rate, Errors, Duration) | Industry standard, covers essential service health |
| Log format | Structured JSON | Machine-parseable, supports log aggregation |
| Tracing | OpenTelemetry | Vendor-neutral, auto-instrumentation, broad ecosystem |
| LLM observability | Langfuse (not LangSmith) | Open-source, self-hosted, built-in prompt management |
| LLM tracing API | @observe(as_type=...) + score_current_span() | v4: semantic types, inline scoring, span filtering |
| Langfuse APIs | Observations API v2 + Metrics API v2 | v4 (Mar 2026): faster querying, aggregations at scale |
| Hook telemetry transport | JSONL under ~/.claude/analytics/, never an SDK in-process | Hooks are per-event processes; SDK init would be paid on every spawn (references/ork-delta.md) |
Detailed Documentation
| Resource | Description |
|---|---|
references/ork-delta.md | Start here. Floors, house decisions and scars that upstream docs do not carry |
references/langfuse-js-v5.md | JS/TS SDK v5 delta from Python 4.x: package map, phantom packages, SpanProcessor vs exporter. Read before writing any JS Langfuse code |
references/experiments-api.md | Langfuse experiments and dataset runs as this repo uses them |
references/evaluation-scores.md | Score shapes and scoring pipeline wiring |
references/session-tracking.md | Session and user grouping across multi-step workflows |
references/metrics-collection.md | Claude Code OTEL metric inventory and collector-side joins |
references/dashboards.md | Dashboard layout conventions |
references/structured-logging.md | Structured log field conventions |
references/dev-agent-lens.md | LiteLLM proxy layer for API-boundary observability |
examples/orchestkit-monitoring-dashboard.md | Worked monitoring dashboard example |
scripts/ | Templates: Prometheus, OpenTelemetry, health checks, Langfuse |
Related Skills
defense-in-depth- Layer 8 observability as part of security architecturedevops-deployment- Observability integration with CI/CD and Kubernetesresilience-patterns- Monitoring circuit breakers and failure scenariosllm-evaluation- Evaluation patterns that integrate with Langfuse scoringcaching- Caching strategies that reduce costs tracked by Langfuse
FAQ
What does the monitoring-observability skill do?
Monitoring and observability patterns for Prometheus metrics, Grafana dashboards, Langfuse v4 LLM tracing (astype, scorecurrentspan, shouldexportspan, LangfuseMedia), and drift detection. Use when adding logging, metrics, distributed tracing, LLM cost tracking, or quality drift monitoring.
How do I install monitoring-observability?
Run npx skills add https://github.com/yonatangross/orchestkit --skill monitoring-observability in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/monitoring-observability/ (personal) or .claude/skills/monitoring-observability/ (project).
Which AI agents can use monitoring-observability?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is monitoring-observability safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains monitoring-observability?
It is published by yonatangross in yonatangross/orchestkit. It has 262 recorded installs.
More from yonatangross/orchestkit
devops-deployment
yonatangross/orchestkit
Use when setting up CI/CD pipelines, containerizing applications, deploying to Kubernetes, or writing infrastructure as code. DevOps & Deployment covers GitHub Actions, Docker, Helm, and Terraform patterns.
responsive-patterns
yonatangross/orchestkit
Responsive design with Container Queries, fluid typography, cqi/cqb units, subgrid, intrinsic layouts, foldable devices, and mobile-first patterns for React applications. Use when building responsive layouts or container queries.
architecture-decision-record
yonatangross/orchestkit
ADR templates in the Nygard format with context, decision, consequences, and alternatives. Use when writing ADRs, recording an architectural decision, or evaluating options.
ui-components
yonatangross/orchestkit
UI component library patterns for shadcn/ui and Radix Primitives. Use when building accessible component libraries, customizing shadcn components, using Radix unstyled primitives, or creating design system foundations.
rag-retrieval
yonatangross/orchestkit
Retrieval-Augmented Generation patterns for grounded LLM responses. Use when building RAG pipelines, embedding documents, implementing hybrid search, contextual retrieval, HyDE, agentic RAG, multimodal RAG, query decomposition, reranking, or pgvector search.
zustand-patterns
yonatangross/orchestkit
Reference for Zustand 5.x state management including slices, middleware, Immer, useShallow, persistence, selectors, and devtools integration. Documents 7 core patterns with TypeScript examples and anti-patterns. Use when building React state management with Zustand instead of Redux.
Popular devops & cloud skills
microsoft-foundry
microsoft/azure-skills
Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent create, managed harness agent create, GitHub Copilot harness agent, create agent, update agent, add tool to agent, invoke agent, agent insights, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure
azure-diagnostics
microsoft/azure-skills
Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.
azure-prepare
microsoft/azure-skills
Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.
azure-ai
microsoft/azure-skills
Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.
azure-deploy
microsoft/azure-skills
Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/deployment-plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.
azure-validate
microsoft/azure-skills
Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity permissions, and prerequisites before deploying. WHEN: validate my app, check deployment readiness, run preflight checks, verify configuration, check if ready to deploy, validate azure.yaml, validate Bicep, test before deploying, troubleshoot deployment errors, validate Azure Functions, validate function app, validate serverless deployment, verify RBAC roles, check role assignments, review managed identity permissions, what-if analysis, validate Container Apps deployment.