Alert Create

Build a SigNoz alert from natural-language intent. Autonomous agents and interactive clients follow the same flow.

Prerequisites

This skill calls SigNoz MCP tools (signoz_create_alert, signoz_list_alert_rules, signoz_get_field_keys, etc.). Confirm they are available; otherwise run signoz-mcp-setup. Never fall back to raw HTTP or fabricate alert configs.

When to use

Use this skill when the user wants to:

• Create, set up, or configure a new alert rule.
• Get paged or notified when a metric, log volume, latency, or error rate crosses a threshold.
• Detect anomalous behavior on a service, host, or signal.
• Catch silent data loss ("alert if data stops arriving from X").

Do NOT use when the user wants to:

• Understand what an existing alert monitors → signoz-explaining-alerts.
• Diagnose why an existing alert fired → signoz-investigating-alerts.
• Modify thresholds, queries, or routing on an existing alert → call signoz_update_alert directly.

Required inputs (strict)

Alert creation writes to a shared system. Enforce this strict input contract; guesses create noisy alerts on the wrong service:

If a required input is missing and undiscoverable, stop before any write and ask through the host's supported clarification UI.

What to include in the question:

• What is missing: name it concretely (e.g. "which resource-attribute filter to use").
• Candidate lists from discovery: concrete values per attribute, e.g.: service.name → frontend, checkout, payments; host.name → prod-api-1, prod-db-1.
• Free-form input so the user can name an unsurfaced value.

In autonomous mode, escalate or use upstream context; never call signoz_create_alert with a guessed value.

Workflow

Step 1: Parse intent and check what's missing

Extract from the user's request:

1. What to monitor: signal type (metrics / logs / traces / exceptions) and the specific condition (CPU, error rate, p99 latency, log count, ...).
2. Resource scope: which service, host, namespace, or environment.
3. Threshold: numeric value and comparison ("above 80%", "below 100/s").
4. Severity: note any severity the user names and any urgency cue ("page me", "just FYI"). Step 5 resolves the final value.
5. Routing: direct channel name(s), or an explicit request to use a confirmed existing org notification policy.

Map signal phrasing to alert type:

The table lists the types the SigNoz alert UI can edit today. Newer types the UI does not surface yet, such as AI_TRACES_BASED_ALERT for LLM and AI-agent span telemetry, are also accepted. Use one only when the user names it or no listed type fits, and say that the UI has no editor for it.

If resource scope is missing, run discovery (Step 2). If still missing after discovery, stop and ask the user (see Required inputs above).

Step 2: Discover resource attributes and metric names

When the user does not name a service / host / namespace, the SigNoz MCP guideline applies: always prefer a resource-attribute filter. Discover candidates instead of guessing:

1. Call signoz_get_field_keys with fieldContext=resource to enumerate resource attributes for the chosen signal.
2. Call signoz_get_field_values for the most likely attribute (typically service.name, then host.name, then k8s.namespace.name) to get concrete values.
3. If the user mentioned a metric by name, call signoz_list_metrics with a search term to verify the exact OTel metric name. Wrong names create alerts that never fire.

Surface the candidates in your clarification request (see Required inputs above). Do not pick one.

Step 3: Check for duplicate alerts

Once the scope is resolved (either provided by the user or discovered in Step 2), check for existing alerts before probing data or authoring a new config: both are wasted work if the user wants to update an existing rule instead.

Call signoz_list_alert_rules and paginate through every page: pagination.hasMore is true until you have walked the full list. This lists configured alert rules (the durable state); do not use signoz_list_alerts, which returns currently triggered/active alert instances and will silently miss rules that are configured but not firing right now. Check for existing rules that match the user's intent (same signal + same scope + similar threshold). If a likely duplicate exists, surface it and ask whether to create a new one anyway, modify the existing one (out of scope here: use signoz_update_alert), or cancel.

Step 4: Probe data existence for the chosen filter (fail fast)

Before authoring any alert config, confirm the specific combination the alert will watch (metric × service × any other filter) actually emits data. The most common silent failure is "metric exists in the catalog and the service exists in the catalog, but the service doesn't emit that metric"; each piece checks out in isolation, the alert saves successfully, and it silently never fires.

Run a single probe over the last 1 hour using the same filter the alert will use, but with the simplest aggregation that confirms data exists:

• Metrics: use signoz_query_metrics with the concrete metricName and the alert's filter; it auto-applies aggregation defaults. If the full v5 tool is necessary, use a metrics aggregation object containing metricName, timeAggregation, and spaceAggregation. Metrics do not accept expression aggregations such as count() or filter-only probes.
• Logs: call signoz_aggregate_logs with aggregation: "count" and the alert filter. count() is a Query Builder expression, not a helper-tool argument.
• Traces: call signoz_aggregate_traces with aggregation: "count" and the alert filter. Do not pass aggregation: "count()".

Inspect the result:

• Probe returns rows → proceed to Step 5.
• Probe returns empty → STOP. Do not build an alert config the user will then be asked to throw away. Stop and ask the user (see Required inputs above), describing what was missing and offering concrete recovery:
  • Service doesn't emit the metric → call signoz_get_field_values signal=metrics name=service.name metricName=<metric> to list the services that do emit it; let the user pick a different service or a different metric.
  • Wrong attribute name (service instead of service.name) → suggest the semantic-convention name and re-probe.
  • Service emits the metric but not in the expected time range → widen the probe window once (e.g. last 24h) before declaring no-data.

Exception: log-based crash / panic / OOMKilled / FATAL alerts. These intentionally have zero matches in a healthy system. The probe will return empty by design. Do not stop; instead, surface the zero-match result and ask the user to confirm before save. Treat this exception narrowly: it applies to "alert me when bad thing happens" log queries, not to alerts that depend on continuous data flow.

Step 5: Build the alert config

Read the signoz://alert/instructions and signoz://alert/examples MCP resources for the canonical, version-current shape.

Compose the full payload; no field is filled in for you. Set these explicitly on every threshold or PromQL rule:

• schemaVersion: "v2alpha1". An omitted schemaVersion is read with v1 semantics and rejected for missing v1 condition fields.
• evaluation: { "kind": "rolling", "spec": { "evalWindow": "5m0s", "frequency": "1m0s" } } unless the intent implies another cadence. Required on v2alpha1 rules.
• notificationSettings: required on v2alpha1 rules; set usePolicy per Step 7 and groupBy on high-cardinality queries.
• labels.severity: the highest tier's severity from Severity defaults below. Without it the rule has no severity in the alert list.
• condition.compositeQuery.panelType: "graph". An omitted value is stored empty.

Anomaly rules use the v1 shape instead: top-level evalWindow and frequency, condition.op / matchType / target / algorithm / seasonality, no schemaVersion, no thresholds, no evaluation, and no notificationSettings.

Threshold/PromQL condition.thresholds requires kind (use "basic") and non-empty spec[]. Current v2 cannot represent alertOnAbsent as the sole trigger. For an exact absent-only request, stop before any write and offer a combined threshold + absence rule only with explicit approval.

For most user intents, the config is one of a small number of patterns:

Threshold op and matchType values. Use these op words: above, below, equal, not_equal, above_or_equal, below_or_equal, and outside_bounds; equals is invalid. Use above for anomaly rules: their absolute score covers spikes and drops.

Conventions this skill writes into the payload (and surfaces in the preview):

• evalWindow: 5m0s, frequency: 1m0s: change only if the intent implies a slower or faster cadence.
• matchType: "on_average" for CPU / memory / latency: smooths transient spikes.
• matchType: "at_least_once" for error counts / error rates: catches any breach.

Severity defaults. Resolve each threshold tier's severity in this order:

1. Take the first matching row below (no cue needed), or warning if none.
2. A user cue overrides the default. A named severity ("critical", "warning at 80%") wins over an urgency cue. "Page me", "wake me up", or "urgent" sets critical. "Just FYI", "no paging", or "low priority" lowers the default one tier (critical to warning, warning to info, info stays info). A cue tied to one threshold ("page me at 90%") applies only to that tier. Ask when urgency cues conflict on the same tier.
3. Name each threshold tier by its severity and set labels.severity to the highest tier, which SigNoz's alert-rule list sorts and filters by. Every firing carries that label, so on a multi-tier rule only threshold.name shows which tier fired. Anomaly rules have no tiers: SigNoz uses the label as the tier name. The Step 8 preview states each tier's severity and why.

Example: "alert me when checkout error rate goes above 2%" is critical (error-rate row); the same request with "just FYI" is warning.

Attribute names: use exact keys returned by signoz_get_field_keys; when available they are usually OTel names such as service.name, not service.

Annotation templates: include moving values; on-call sees the notification, not the config:

• summary: single-line headline. Include the resource scope and the numeric value: "checkoutservice error rate {{$value}}% above 3%".
• description: longer message. Include {{$value}}, {{$threshold}}, the groupBy values (e.g. {{$labels.service_name}}), and a sentence on what to do or where to look. For count-based alerts include the count explicitly: "{{$value}} crash log lines in the last 5 minutes from service {{$labels.service_name}}".

Use {{$value}} for the breaching value, {{$threshold}} for the target, and {{$labels.<key>}} for groupBy values (note SigNoz substitutes the dotted attribute name with underscores: service.name → service_name).

Common query shapes: conventions

Read signoz://alert/examples for the authoritative JSON of all patterns (error rate, p99 latency, log volume, absent-data, anomaly, PromQL, ClickHouse SQL). The conventions that don't live in the schema:

• Error-rate formula: set disabled: true on the component queries A and B so only the formula F1 renders in the alert chart and notification. The raw counts are intermediate, not the alert signal: forgetting this clutters the preview with three series and confuses the on-call engineer reading the notification.
• p99 latency: the query emits nanoseconds, but express the threshold in the user's unit (for example target: 2, targetUnit: "s"); SigNoz converts it during evaluation.
• Low-traffic percentile guard: put count() > N in the percentile query's having.expression and set stepInterval to the requested bucket size (for example, 60 for “per minute”). Do not invent comparison operators inside a formula such as A * (B >= N).
• Log volume spike: prefer groupBy: service.name over a hard filter when the user said "any service": groupBy provides the scoping AND keeps the notification useful per-service.

Step 6: Dry-run the full query and validate the threshold

Step 4 confirmed data flows. Step 6 does two things:

1. Validate query shape. Run the full builder spec (with groupBy, formulas, disabled component queries, and non-string filters): Step 4's bare count() probe doesn't exercise these. The create-alert schema accepts queries that error at evaluation (numeric groupBy, unquoted bool filter, mismatched aggregation). Any HTTP 5xx or "filter type mismatch" = fix the config before proceeding to (2). disabled: true on formula component queries (A, B in A * 100 / B) is the recommended pattern, not a failure (see Step 5).
2. Calibrate the threshold. Given the validated query, would the alert have fired a sensible number of times in the last hour?

Run the full primary query (or formula) over the last hour:

• signoz_execute_builder_query for all builder, formula, and PromQL queries: set compositeQuery.queries[].type to builder_query / builder_formula / promql as appropriate. Alert PromQL specs carry only name / query / legend / disabled; dry-run execution PromQL specs may also carry step / stats. Still omit builder-only stepInterval. Put the string in spec.query, read signoz://promql/instructions for the UTF-8 quoted-selector form SigNoz requires ({"metric.name.with.dots"}, not the underscored or bare-dotted forms), and keep alert PromQL fully literal: no $var, $__rate_interval, or other dashboard variable is evaluated.
• Alert specs omit time bounds, but this dry-run cannot: set outer-query start / end to absolute JSON integer Unix-ms (e.g. now−3600000 → now), or signoz_execute_builder_query fails with missing start or end timestamp.
• For intent grouped by a dimension, each execution groupBy[].name is the exact Step 2 key (e.g. k8s.pod.name), never empty. Omit groupBy otherwise.
• signoz_aggregate_logs / signoz_aggregate_traces when those fit better.
• signoz_query_metrics when the alert query targets a single known metric by metricName: the tool auto-applies aggregation defaults and accepts filter, groupBy, and formula alongside. PromQL is not supported here; use signoz_execute_builder_query for that.

Omit limit and order from every query in the persisted rule, builder queries, formula inputs, and formula results alike. On a grouped alert query limit plus order act as a top-N series cap: only the top-N groups ranked over the whole evaluation window are evaluated, so series beyond N are never evaluated, a group with a short-lived spike can be ranked out, and a below alert ordered descending can exclude every breaching series. Set them only when top-N alerting is the explicit intent ("alert on the five busiest hosts"), using Query Builder v5 wire order, never dashboard orderBy.

The dry-run is different: signoz_execute_builder_query is a display context with a response budget, so the bounds signoz-generating-queries teaches for it (a positive limit and v5 order on each query) are fine there. They stay in the dry-run envelope only; never copy them into the signoz_create_alert payload. Because the dry-run returns at most limit groups, a breach count computed from it covers the returned groups only; when the groupBy can exceed that bound, say so in the preview rather than reporting the count as complete.

Compute how many evaluation points breached the proposed threshold. Surface in the preview as "would have fired N times in the last 1h". A 1h window is too short to grade most alerts: only the upper extreme is actionable:

• N is large (e.g. > 30) → likely alert storm. Surface and recommend tightening or adding hysteresis (recoveryTarget).
• N = 0 → expected for a healthy system; do not nudge the user to loosen. Only flag if the user said they'd expect the alert firing right now (e.g. during an active incident).
• N is small and non-zero → report the count; the user decides whether the threshold is right. One hour can't distinguish "tuned well" from "barely caught a transient".

3. Exceptions:
   • Anomaly alerts: execute the underlying metric query without the anomaly functions transform to verify its shape and data, then skip the breach count (Z-scores aren't directly comparable to raw values). Be explicit that this validates the base query, not anomaly scoring.
   • Log-based crash / panic / OOMKilled / FATAL alerts: these intentionally have zero matches in a healthy system. Step 4 has already surfaced the zero-match result and obtained user confirmation; skip the breach count.

If Step 4 was somehow skipped (e.g. a downstream skill is invoking this flow mid-stream), the no-data stop rule applies here too: empty result → stop and ask the user (see Required inputs above) instead of saving an alert that will never fire.

Step 7: Resolve notification routing

Choose direct or org-policy routing after dry-run and final severity. A missing channel does not authorize policy routing.

For a confirmed existing org policy on a threshold/PromQL rule, set notificationSettings.usePolicy: true, preserve the confirmed matcher labels and threshold tier, and omit threshold channels and preferredChannels. This skill does not create policies. If the policy or match criteria are not confirmed, stop and ask. Anomaly rules are direct-only. If the criteria route tiers by severity, stop and ask: every tier matches as the highest (Step 5). Offer a threshold.name matcher, one rule per tier, or direct routing.

For direct routing, reuse a fully paginated same-operation channel result or page signoz_list_notification_channels until hasMore=false; its total counts filtered matches. Match and route by immutable displayName, not the machine name; show displayName and kind when asking the user to pick. Offer creation only from user-provided config and never create automatically. Build signoz_create_notification_channel arguments from its current input schema, which defines the supported provider kinds and identity fields: provider settings go inside config, never as flat fields. Ask whether the user wants a test notification and send test: true only if they agree. On PERMISSION_DENIED, use an admin-created channel or a short-lived minimum-role credential from the host's secret store. If routing remains unresolved, stop and ask.

Put exact returned display names in each threshold's channels; anomaly rules use top-level preferredChannels. Never substitute one routing field for the other. A threshold tier with no channels, or a displayName that does not exist, fails unless notificationSettings.usePolicy is true. Top-level preferredChannels on a v2alpha1 rule is stored unused, so a threshold rule routed only through it saves and notifies nobody.

Updating or handling secret-bearing channel config

signoz_update_notification_channel is full replacement: resolve id, get the channel, copy complete config, change requested fields, then send id plus full config. Do not send or change name or displayName. Preserve credentials and effective sendResolved without echoing them. A kind change needs the complete new provider config. Update tests require explicit opt-in.

If a requested test fails after a successful write, report mutation and test status separately. On a post-write auth error with mutationCommitted: true, retain the id and inspect after authentication; do not replay mutation or test. Transport success does not prove final provider delivery.

Slack webhook URLs, PagerDuty integration keys, and similar webhook tokens are secrets. When the user supplies them inline, treat them as opaque inputs and follow these rules:

• Do not echo the secret back. Never include the webhook URL, integration key, or any password-like token in chat output, previews, confirmation messages, summaries, or the <navigation_suggestions> payload. Refer to the channel by its displayName only ("Slack channel Platform Alerts created") and omit the value entirely.
• Do not stash secrets in clarification context. If you need to ask the user a follow-up question after they pasted a secret, do not include the secret value in the clarification message, discovered_context, or any other field that the host may persist for resume. Refer to it symbolically (e.g. "the webhook you just provided").
• One-pass only. Pass the secret directly to signoz_create_notification_channel and do not retain it in any intermediate prose. After the create call succeeds, refer to the channel by display name; after a failure, ask the user to re-paste rather than echoing what they sent.
• If the user instead asks "how do I set up a Slack channel?": that is a docs question, not a create-channel request. Answer with the docs flow (the SigNoz UI's Notification Channels page) and do not solicit the secret in chat at all. Prefer the UI path when the user seems uncertain about exposing the token.

Step 8: Preview the prepared config

Emit a one-paragraph plain-language summary of what will be created: no raw JSON dump. The user-facing facts (what fires, on what scope, at what threshold, where it routes) are captured by the summary; clicking through the JSON does not catch query-shape errors (Step 6's dry-run does).

Summary: This alert fires when [condition] for [resource scope], evaluated every [frequency] over the last [window]. Severity: [tier], because [table row or user cue]. Thresholds: warning at X, critical at Y. Notifications route through [direct channels / confirmed org policy]. Dry-run on the last hour: would have fired N times.

Step 9: Save and report

1. Call signoz_create_alert with the config from Step 8.
2. Name collision: if signoz_create_alert returns a duplicate-name error, do not suffix-append or call signoz_update_alert. Stop and tell the user the existing alert blocked creation; offer to use a different name or modify the existing alert (which is out of scope for this skill).
3. On success, report:
   • The alert ID and name.
   • What it watches, at what threshold, and at what severity.
   • How notifications route (direct channels or the confirmed org policy).
   • The dry-run summary ("would have fired N times in last 1h").

Guardrails

• Strict inputs over guessing Resource attribute and notification routing are required. Direct routing needs a channel; policy routing needs a confirmed existing match and does not require a direct channel.
• Always paginate signoz_list_alert_rules Stopping at page 1 misses duplicates and produces noise.
• Dry-run is mandatory Complete Steps 4 and 6 before signoz_create_alert; a never-firing alert creates false confidence.
• Threshold operators use canonical words Use the op and matchType words from Step 5, never equals.
• Compose the full payload Set schemaVersion, evaluation, notificationSettings, labels.severity, and panelType yourself (Step 5); nothing fills them in.
• No limit or order on persisted alert queries They cap a grouped alert to its top-N series (Step 6). Keep them in the dry-run envelope only.
• Signal must match alertType signal: "logs" requires LOGS_BASED_ALERT. Mismatches fail validation.
• Anomaly rules are metrics-only anomaly_rule + non-metric alertType is rejected.
• Routing must be confirmed. Direct routes use exact returned displayName values from the fully paginated signoz_list_notification_channels; put them in per-threshold channels for threshold/PromQL rules or top-level preferredChannels for anomaly rules. Confirmed policy routes set notificationSettings.usePolicy and omit direct channel fields.
• Never echo channel secrets. Slack webhook URLs, PagerDuty integration keys, and similar webhook tokens are secrets. Pass them to signoz_create_notification_channel once and never repeat the value in chat output, previews, confirmations, summaries, clarification payloads, or navigation suggestions. Refer to the channel by display name only after creation; ask the user to re-paste on failure rather than reproducing what they sent.

Examples

Four canonical alert flows (multi-severity metric threshold, error-rate formula, log-volume groupBy, and anomaly detection) live in references/examples.md.

Additional resources

• signoz://alert/instructions and signoz://alert/examples MCP resources: the alert payload shape, the fields to set explicitly, threshold codes, filter expression syntax, and version-current pattern examples. Always preferred over any transcribed copy.
• signoz-generating-queries skill: for authoring PromQL or testing queries before wrapping them in an alert.