observability-fundamentals
First principles of observability: wide events, high cardinality, the core analysis loop, events vs metrics vs logs, and how instrumentation connects to debugging. Trigger phrases: "what is observability", "why observability", "why Honeycomb", "events vs metrics vs logs", "why wide events", "what is high cardinality", "core analysis loop", "observability vs monitoring", "what is dimensionality", "explain observability".
- Installs
- 54
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/honeycombio/agent-skill --skill observability-fundamentals- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads observability-fundamentals automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Observability Fundamentals
First principles behind Honeycomb's approach to observability. Use this to ground recommendations and answer conceptual questions — for SDK setup and tool-specific guidance, see the otel-instrumentation and query-patterns skills.
Definitions
Observability: The ability to understand and explain any state your system can get into, no matter how novel or complex — by examining what the system produces, without deploying new code for each new question.
Wide event: A flat key-value record capturing the full context of a unit of work — who made the request, which endpoint, cache hit/miss, build version, duration, error status, and any business context relevant to the operation. In OpenTelemetry, a span is a wide event.
High cardinality: The number of unique values a field can have. user.id with
millions of values is high cardinality. http.method with a handful is low cardinality.
High dimensionality: The number of distinct fields on your events. A span with 50 attributes has high dimensionality.
| Concept | Observability | Traditional Monitoring |
|---|---|---|
| Questions | Arbitrary, unknown ahead of time | Pre-defined (dashboards, alerts) |
| Data shape | Decided at query time | Decided at instrumentation time |
| Cardinality | High cardinality is valuable | High cardinality is expensive |
| Investigation | Explore → narrow → confirm | Check dashboard → escalate |
Why Wide Events
The shape of the data you collect constrains the questions you can ask later. Metrics pre-aggregate context away at instrumentation time. Wide events preserve context and let you decide the shape of your analysis at query time.
Every attribute on a span is a queryable dimension. Adding user.id, deployment.version,
and cache.hit to the same span lets you correlate them in a single query — "slow
requests are from tenant X on version 2.3.1 with cache misses." Separate metrics can't
do this because each dimension combination creates a new time series.
Honeycomb's storage engine handles high cardinality and dimensionality without the
cost explosion that affects metrics systems. Adding a high-cardinality field like
user.id doesn't create millions of time series — it's another column on each event,
aggregated at query time.
Events vs Metrics vs Logs
| Structured Events (Spans) | Metrics | Logs | |
|---|---|---|---|
| Captures | Full request context (all attributes) | Pre-aggregated numbers with low-cardinality tags | Text or structured fields per line |
| Discards | Nothing — raw events retained | Individual requests, high-cardinality dimensions | Correlation across lines (without trace context) |
| Query power | GROUP BY, filter, BubbleUp on any dimension | Fast aggregates on pre-defined dimensions | Text search, structured field queries |
| Cost scaling | Linear with event volume | Exponential with dimension count (cardinality) | Linear with volume, query cost varies |
| Best for | Investigation, root cause analysis | Cheap alerting, long-term trends | Audit trails, rare events |
The same instrumentation effort that produces a metric or log line can produce a wide event — and the event gives you all three capabilities: count it (metric), read it (log), analyze it across dimensions (observability).
For code examples showing the same operation instrumented three ways, see
${CLAUDE_PLUGIN_ROOT}/skills/observability-fundamentals/references/events-vs-metrics-vs-logs.md.
The Core Analysis Loop
Debugging in Honeycomb follows a loop: Define → Visualize → Investigate → Evaluate.
- Define — Frame the question. Start from an alert, SLO budget burn, or user report.
- Visualize — Run a query to see the shape of the problem (HEATMAP, COUNT, P99).
- Investigate — Narrow down with BubbleUp (automated outlier-vs-baseline comparison across all dimensions) and trace analysis.
- Evaluate — Confirm the hypothesis by querying with and without the suspected cause.
Then loop — each answer raises new questions. BubbleUp automates steps 2-3 by comparing distributions across every column, but it only works if events have enough dimensions to diff on.
For the structured workflow that implements this loop with Honeycomb's tools, see the production-investigation skill.
Instrumentation Connects to Investigation
Every attribute on a span is a dimension BubbleUp can use to find root causes. The attributes that matter most during incidents answer three questions:
- Who is affected? — user, tenant, account tier, region
- What changed? — deployment version, feature flag, config version
- Where is the bottleneck? — business operation spans, timing breakdowns, cache state
Instrument for the questions you'll ask at 3am, not for completeness. If BubbleUp returns nothing useful during an investigation, the issue is usually an instrumentation gap — add the missing dimensions and try again.
For the complete attribute catalog, see
${CLAUDE_PLUGIN_ROOT}/skills/otel-instrumentation/references/wide-event-attributes.md.
For SDK guidance on adding attributes, see the otel-instrumentation skill.
Instrumentation as a Development Practice
Instrumentation is not a one-time setup task. The engineers who write the code are best positioned to know which operations are critical, which paths are error-prone, and what context helps during debugging. Treat instrumentation like testing: plan telemetry when planning features, review it in code reviews, and add missing dimensions as post-incident follow-ups.
Additional Resources
Reference Files
${CLAUDE_PLUGIN_ROOT}/skills/observability-fundamentals/references/events-vs-metrics-vs-logs.md— Code examples: same operation as event, metric, and log
Cross-References
- For SDK setup and custom instrumentation: otel-instrumentation skill
- For the investigation workflow implementing the core analysis loop: production-investigation skill
- For autonomous instrumentation gap analysis: instrumentation-advisor agent
FAQ
What does the observability-fundamentals skill do?
First principles of observability: wide events, high cardinality, the core analysis loop, events vs metrics vs logs, and how instrumentation connects to debugging. Trigger phrases: "what is observability", "why observability", "why Honeycomb", "events vs metrics vs logs", "why wide events", "what is high cardinality", "core analysis loop", "observability vs monitoring", "what is dimensionality", "explain observability".
How do I install observability-fundamentals?
Run npx skills add https://github.com/honeycombio/agent-skill --skill observability-fundamentals in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/observability-fundamentals/ (personal) or .claude/skills/observability-fundamentals/ (project).
Which AI agents can use observability-fundamentals?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is observability-fundamentals safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains observability-fundamentals?
It is published by honeycombio in honeycombio/agent-skill. It has 54 recorded installs.
More from honeycombio/agent-skill
otel-instrumentation
honeycombio/agent-skill
OpenTelemetry SDK setup, custom instrumentation, and sending data to Honeycomb. Trigger phrases: "instrument my app", "add tracing", "set up OpenTelemetry", "configure OTel", "add custom spans", "add attributes to spans", "send traces to Honeycomb", "set up OTLP", "configure sampling", "add span events", "add span links", "set up tracing for [any language]", "configure the OTel Collector".
query-patterns
honeycombio/agent-skill
Constructing and interpreting Honeycomb queries on trace, log and event datasets: operation selection (percentiles not AVG, HEATMAP for distributions), relational fields (root., parent., any., none.), calculated fields, query math, and reading results (P99/P50 ratios, heatmap bands, TOTAL/OTHER rows, raw JSON via queryresultjson). Trigger phrases: "show me latency", "error rate", "find slow requests", "find outliers", "interpret results", "relational fields", "calculated fields", "download raw results". For metrics datasets (datasettype=metrics) use metrics-queries.
create-honeycomb-board
honeycombio/agent-skill
Design and create a Honeycomb board (dashboard) with queries and SLOs. Trigger phrases: "make a board", "build a dashboard", "create a Honeycomb board", "make a dashboard in Honeycomb", "set up a board", "dashboard for my service", "visualize service health", "golden signals dashboard", "set up monitoring board".
production-investigation
honeycombio/agent-skill
Workflows for investigating production issues in Honeycomb: context priming, broad query, BubbleUp, trace analysis, verification, and chaining results between steps to a root cause. Trigger phrases: "investigate production issue", "debug latency spike", "find root cause", "use BubbleUp", "analyze traces", "debug an outage", "why is my API slow", "errors are increasing", "health check", "SLO burning".
otel-migration
honeycombio/agent-skill
Retrofitting OpenTelemetry into an existing, uninstrumented application. Trigger phrases: "migrate existing app to OTel", "add OpenTelemetry to existing project", "retrofit OTel into my codebase", "thread context through my code", "context propagation", "bridge Prometheus metrics to OTel", "logging bridge", "migrate logging to OTel", "slog bridge", "logback bridge", "verify my instrumentation", "traces are disconnected", "orphaned spans", "migrate to OpenTelemetry", "OTel migration plan", "how do I sequence an OTel migration", "add tracing to existing code", "refactor for context propagation", "Fiber context gotcha", "keep existing logging working with OTel", "add OTel without breaking Prometheus", "bridge existing metrics", "coexist with existing monitoring". For greenfield instrumentation use otel-instrumentation; for Beelines use beeline-migration.
Popular devops & cloud skills
microsoft-foundry
microsoft/azure-skills
Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent create, managed harness agent create, GitHub Copilot harness agent, create agent, update agent, add tool to agent, invoke agent, agent insights, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure
azure-diagnostics
microsoft/azure-skills
Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.
azure-prepare
microsoft/azure-skills
Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.
azure-ai
microsoft/azure-skills
Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.
azure-deploy
microsoft/azure-skills
Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/deployment-plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.
azure-validate
microsoft/azure-skills
Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity permissions, and prerequisites before deploying. WHEN: validate my app, check deployment readiness, run preflight checks, verify configuration, check if ready to deploy, validate azure.yaml, validate Bicep, test before deploying, troubleshoot deployment errors, validate Azure Functions, validate function app, validate serverless deployment, verify RBAC roles, check role assignments, review managed identity permissions, what-if analysis, validate Container Apps deployment.