synthetic-monitoring

Quick Route

Discovery Questions

Check .agents/qa-project-context.md first. If it exists, use it as context and skip questions already answered there. Each answer changes the probe set, the alerting config, or the SLA math.

Critical flows (decides which probes you write):

• What are the 5-10 most important user journeys (login, search, checkout, signup, core workflow)? These become your probe list.
• Which flows, if broken, cause immediate revenue loss or churn? These get the shortest frequency and page on-call.
• Are there flows that break silently (data sync, background jobs, webhooks)? Silent failures need probes most — no user reports them.
• Do you have documented SLAs/SLOs for availability and response time? They set the downtime budget and the alert thresholds.

Current monitoring (decides where the gaps are):

• What exists today (uptime checks, APM, error tracking, dashboards)? Avoid duplicating; find the gap.
• Are there gaps between what monitoring covers and what users experience? That gap is where probes earn their keep.
• How do you learn about production issues today (alerts, user reports, social media)? If it's user reports, detection time is your first metric to fix.
• What was the last outage, and how long before it was detected? Sets the detection-time target probes must beat.

Infrastructure (decides regions and CDN assertions):

• Is the app served from multiple regions or one? Multi-region serving needs multi-region probing.
• Are there CDN, caching, or edge layers that could mask origin failures? If yes, probes must assert on x-cache/origin headers.
• Do third-party dependencies (payment, auth, email) have their own monitoring? Their status pages feed your runbooks.
• What alerting systems exist (PagerDuty, OpsGenie, Slack, email)? Decides the routing config.

Test accounts (decides data safety):

• Do dedicated synthetic test accounts exist in production? Without them, probes pollute real data and analytics.
• Can test accounts be excluded from analytics, billing, and email campaigns? If not, probe traffic skews every downstream number.
• Is there API access for programmatic account setup and data cleanup? Decides whether create-then-delete probes are viable.

Core Principles

1. Synthetic tests validate the user experience continuously

RUM tells you what happened; synthetic tells you what is happening right now, whether or not real users are active. At 3 AM when traffic is zero, synthetic probes are the only thing checking the app works. Complement RUM, never replace it: synthetic covers known paths with predictable inputs, RUM discovers the creative ways real users break things.

2. Keep probes simple and fast

A probe that takes 2 minutes and touches 15 pages is not a probe — it is an E2E suite running in production. Probes are short (per-probe wall-clock budget under 30 seconds), focused (one critical path each), and stable (retries: 0, zero flakiness tolerance). The 30-second ceiling is a hard budget: a slow probe that "still passes" is masking a degradation users feel.

3. Alert on trends, not single failures

A single probe failure is noise — network blips and DNS hiccups cause them constantly. Two consecutive failures are a signal; failures from 2+ regions confirm it is not local. Configure consecutive-failure and multi-region thresholds, or alert fatigue teaches the team to ignore the pager.

4. Probes must not affect production data

Probes run every few minutes, 24/7. Even small side effects (a created record, an incremented counter) compound. Probes must be non-destructive, created data cleaned up immediately, and synthetic traffic excluded from analytics, billing, and — critically — from SLO/error-budget math itself, or probes inflate your own reliability numbers.

5. Assert on the goal, not the status code

A login page that returns 200 but never authenticates is broken. A search page that returns 200 with zero results is broken. Probes assert that the user can accomplish their goal — data loads, auth succeeds, results appear — not merely that the page returns a 2xx.

Probe Design

Design probes around critical user journeys, not infrastructure components. Users do not care if your load balancer is healthy — they care if they can log in and use the product.

Probe implementations

Three probe shapes cover most needs: a login flow (Playwright browser), an API health check (status + auth + latency budget), and a search probe (fill query → submit → assert on results, not just status). Keep each to one critical path, a tight timeout, and retries: 0. See references/probe-implementations.md for runnable login-flow, API-health, and search probes plus the environment-aware config.

Non-destructive probe patterns

Safe patterns:
  - Read-only operations: GET requests, page loads, searches
  - Sandbox transactions: payment in test mode, email to internal addresses
  - Create-then-delete: create a draft, verify, delete immediately
  - Create-and-own: when the system creates but cannot delete (orders, tickets), a
    dedicated account owns the records, nothing downstream is triggered (no payment
    event, so nothing ships), every run writes a ledger of the ids it created, and a
    purge the team signed off runs on a schedule. Say what accumulates: 144 pending
    orders a day at a 10-minute cadence. Do not drop the journey probe because the
    cleanup is manual; say so in the plan.
  - Synthetic flag: operations tagged as synthetic, excluded from processing

Dangerous patterns (avoid):
  - Creating real orders, tickets, or user-facing records
  - Triggering real notifications (email, SMS, push)
  - Modifying shared resources (config, permissions, settings)
  - Operations that cannot be automatically cleaned up

Dedicated test accounts

Account requirements:
  - Clearly identifiable: email contains "synthetic" or "monitor"
  - Flagged in database with an is_synthetic flag (is_synthetic = true)
  - Excluded from analytics, billing, email campaigns, support queues
  - Excluded from RUM and SLO/error-budget pipelines (synthetic traffic is not real traffic)
  - Pre-populated with stable test data that probes can rely on
  - Credentials in a secrets manager (process.env), rotated quarterly
  - Separate accounts per concurrent probe (avoid state conflicts); with fewer accounts
    than probes, share one between read-only probes and keep the writing probe alone

Platform Options

Probes can be authored in Playwright (TS/JS), Puppeteer, k6 (JS — k6 2.0 shipped May 2026 with AI-assisted test authoring and a clearer Assertions API; first-class for synthetic), or Python (CloudWatch Synthetics, Checkly). Pick what your team already maintains. Avoid: Pingdom for new setups — it is a legacy uptime tool; prefer code-first alternatives (Checkly, Grafana, custom Playwright) that version-control probes alongside your app.

Custom and managed implementations

Self-managed: schedule Playwright probes with a GitHub Actions schedule cron (every 5 minutes), inject prod credentials via secrets, and report results to a monitoring webhook. Managed: Checkly is Playwright-native and runs from multiple locations on a fixed frequency. Either way, make probes environment-aware so the same code runs against staging and production with different base URLs and thresholds. See references/platforms-and-ci.md for the GitHub Actions workflow, the Checkly config, the alert-routing rules, and the runbook template.

Alerting Integration

Not every probe failure is an incident. Configure rules that cut noise while catching real problems.

Alerting rules:
  - Single failure: log, do not alert (transient network issue)
  - 2 consecutive failures from same region: warning (possible issue)
  - 2 consecutive failures from 2+ regions: alert on-call (confirmed outage)
  - Latency >2x baseline for 10 minutes: warning (performance degradation)
  - Latency >3x baseline for 5 minutes: alert on-call (severe degradation)
  - Any probe timeout: alert if 3 consecutive (service unresponsive)

Routing. Critical failures on revenue paths (login, checkout, api-health) page on-call (PagerDuty + Slack incidents) with a short repeat interval; warnings on secondary probes go to a Slack monitoring channel; info-level events use a long repeat interval. The probe must tag each result with a severity and probe label for these routes to match — see the routing note in references/platforms-and-ci.md for the full alerting-rules.yaml and the tagging step. Severity comes from the consecutive-failure count, so the reporter derives it, not the probe. Before a reporter or alert manager exists, tag each result with probe and alert-route annotations, attach the probe's timings, and write the derivation rule into the monitoring plan so the first reporter implements it rather than inventing it.

Suppress synthetic alerts during planned maintenance. A maintenance window should silence synthetic paging (the probes will fail by design) and exclude that window from error-budget math, or scheduled work burns budget and pages on-call for nothing.

Alert message format — include enough context to start investigating immediately:

Alert template:
  Title: [SYNTHETIC] {probe_name} failing from {region}
  Severity: {critical|warning|info}
  Consecutive failures: {count}
  Last success: {timestamp}
  Error: {error_message}
  Duration: {last_response_time_ms}ms (threshold: {threshold}ms)
  Dashboard: {link_to_dashboard}
  Runbook: {link_to_runbook}
  Regions affected: {list_of_failing_regions}

The {link_to_runbook} points at a per-probe runbook (six lines: what it tests, first checks, manual repro, escalation, dashboard, owner). See the template in references/platforms-and-ci.md.

SLA Validation

Availability calculation

Availability = (total_minutes - downtime_minutes) / total_minutes × 100

Where:
  - total_minutes = calendar month in minutes (43,200 for a 30-day month)
  - downtime_minutes = minutes where synthetic probes detected failure

SLA tiers (downtime/month, monthly basis on 43,200 min):
  99.0%  = 432 min   = 7h 12min     (basic web app)
  99.9%  = 43.2 min  = 43min 12s    (business application)
  99.95% = 21.6 min  = 21min 36s    (critical SaaS)
  99.99% = 4.32 min  = 4min 19s     (infrastructure/platform)

These are common availability targets, not prescriptive tiers. Pick targets from a user-impact analysis, not by tier name. Modern practice (Google SRE Workbook, OpenSLO) favors explicit SLO + error-budget policies over labelled tiers — define what user-visible failure looks like, set the budget user impact tolerates, and let the SLO follow. References: https://sre.google/workbook/ ; https://openslo.com/

Response time percentiles

Track percentiles, not averages — averages hide the worst experiences.

SLA response time targets (example):
  Homepage load:  P50 < 1s,    P95 < 3s,   P99 < 5s
  API response:   P50 < 200ms, P95 < 500ms, P99 < 1s
  Search results: P50 < 500ms, P95 < 2s,   P99 < 4s
  Login flow:     P50 < 2s,    P95 < 5s,   P99 < 8s

Error budget tracking

Error budget connects SLA targets to engineering decisions.

Error budget calculation:
  SLO: 99.9% availability on a 43,200-min month
  Budget: 0.1% of total time = 43.2 minutes/month

  Budget consumed this month: 12 minutes (28%)
  Budget remaining: 31.2 minutes (72%)

Actions by budget status:
  >50% remaining: normal operations, ship features
  25-50% remaining: caution, review recent changes
  <25% remaining: freeze non-critical deploys, focus on reliability
  Budget exhausted: incident mode, every deploy needs extra scrutiny

Exclude synthetic-probe downtime caused by your own maintenance windows from this calculation, or planned work shows as budget burn.

Multi-Region Monitoring

Run probes from regions where your users are. A service that works from us-east-1 but is broken from ap-southeast-1 is broken for APAC users.

Region selection strategy:
  - Minimum 3 regions for global services
  - Always include: closest to primary infrastructure, largest user base, farthest from primary
  - Example for US-primary service: us-east-1, eu-west-1, ap-southeast-1
  - Example for EU-primary service: eu-west-1, us-east-1, ap-northeast-1

Track regional latency separately — a global average hides regional degradation.

Regional latency dashboard:
  Region       | P50    | P95    | Status
  us-east-1    | 120ms  | 340ms  | healthy
  eu-west-1    | 280ms  | 620ms  | healthy
  ap-southeast | 450ms  | 1200ms | warning (P95 above threshold)

Alert when:
  - Any region's P95 exceeds its regional threshold
  - Latency difference between regions exceeds 5x (CDN or routing issue)
  - A region that was healthy becomes consistently degraded

CDN validation. Synthetic probes verify CDN caching by checking response headers (x-cache, cf-cache-status) for HIT and confirming the server header matches the expected provider. This catches CDN misconfigurations — and origin failures masked by a stale cache — before users hit slow uncached responses.

Anti-Patterns

Complex probes that break often

A probe that navigates 10 pages, fills 5 forms, and asserts on 20 elements is an E2E test, not a synthetic probe. When it breaks, you cannot tell if the app is down or the probe is flaky. Fix: one critical path per probe, under 30 seconds, under 5 assertions. A failure should make it immediately clear what is broken.

Alerting on every single failure

Network blips, DNS hiccups, and transient cloud issues cause occasional failures. Alerting on every one produces noise that teaches the team to ignore alerts. Fix: require 2-3 consecutive failures and failures from 2+ regions before paging. Escalating severity: first failure logs, second warns, third pages.

No test account isolation

Probes sharing one account interfere — one probe changes a setting, another fails because it expected the default. Fix: one dedicated synthetic account per concurrent probe, flagged synthetic, excluded from analytics and billing.

Monitoring only the happy path

Probes that only check "page loads, returns 200" miss broken functionality behind a loading page. A login that returns 200 but never authenticates is not working. Fix: assert on meaningful content — data loads, auth succeeds, the core action completes, search returns results. One "can the user accomplish their goal" probe is worth ten "does the page return 200" probes.

No runbook for probe failures

An alert fires at 3 AM. On-call sees "Login probe failing" but has no idea what to check first, what the probe does, or how to tell a real outage from a probe issue. Fix: every probe links a runbook (what it tests; first checks — third-party status, recent deploys, app telemetry; manual repro; escalation; dashboard link). See the template in references/platforms-and-ci.md.

Emerging pattern (2026): agent-driven first response. Tools like Checkly Rocky (GA 2026, automated root-cause analysis across check types) and Honeycomb Canvas Skills (Agent Observability, launched May 2026; open-source honeycombio/agent-skill repo ships a honeycomb-investigator and instrumentation-advisor for Claude Code and Cursor) read probe context + linked runbook + telemetry and post a candidate diagnosis to chat. Treat this as triage assist — it shortens MTTR for routine failures, but it does not replace on-call judgment for novel incidents.

Failure Modes

Verification

Prove the monitoring works before trusting it — smallest check first.

1. Probes pass against staging. Run npx playwright test probes/ --reporter=list (or the project's own test command when it pins a config and reporters) against staging twice and confirm every probe passes within its declared timeout (no probe exceeds the 30s ceiling).
2. A failure actually pages. Point one probe at a known-bad URL (a 503 or a wrong path), let it run the configured consecutive-failure count, and confirm the alert routes to on-call within the SLA detection time — and resolves when you revert.
3. Assertions are meaningful, not status-only. Temporarily break the asserted content (rename the dashboard heading on staging) and confirm the probe fails. A probe that stays green here is asserting on the wrong thing.
4. Synthetic traffic is excluded. Query the analytics/RUM/billing pipeline for is_synthetic traffic and confirm it is filtered out.

Done When

• A probe exists for every critical user journey identified in discovery (not only homepage uptime or a single health endpoint), and each asserts on the goal, not just a 2xx.
• Each probe's wall-clock budget is under 30s, with retries: 0.
• Alerting config encodes a ≥2 consecutive-failure rule AND a multi-region confirmation rule, and probe results are tagged with severity + probe so routes match. A single-region or loopback deployment records the region rule as not applicable, not as done.
• The SLA dashboard shows current availability and error-budget consumption, and synthetic traffic is excluded from RUM, billing, and error-budget pipelines.
• A test alert (probe pointed at a failing URL) reaches on-call within the SLA detection time — confirmed once, not assumed.
• Every probe links a non-empty runbook with first checks, manual repro, and an escalation path.
• A scheduled review job or recurring calendar item for monitoring health exists (not just an intention to review).

Reference Files (in references/)

• probe-implementations.md — Runnable login-flow, API-health, and search probes, plus the environment-aware probe config.
• platforms-and-ci.md — GitHub Actions scheduling workflow, Checkly config, alert-routing rules + tagging note, and the per-probe runbook template.

Related Skills

• testing-in-production — safe-release techniques (flags, canary, guardrail metrics) applied during a rollout; synthetic monitoring is the schedule-driven validation that runs continuously after.
• release-readiness — a one-shot post-deploy smoke gate tied to a specific release lives there; come here for the continuous, schedule-driven version that keeps running long after the release.
• observability-driven-testing — uses production telemetry (including signals from these probes) as input to design new tests; this skill instead produces the probes and their telemetry.
• performance-testing — load tests measure capacity on demand; synthetic probes track production performance trends between those load tests.
• qa-metrics — availability, response-time percentiles, and error-budget consumption from probes feed the quality dashboards defined there.
• ci-cd-integration — go there to wire synthetic probes into a pipeline as a post-deploy verification stage.