dd-audit-cost-spike-investigation
Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when/what spiked) with Audit Trail config changes (who changed what in the preceding window).
- Installs
- 476
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/datadog-labs/agent-skills --skill dd-audit-cost-spike-investigation- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads dd-audit-cost-spike-investigation automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Audit Trail: Cost / Usage Spike Investigation
Identify what caused a Datadog usage spike by correlating billing data with configuration change history.
The causal chain is: someone changed something → that change increased data volume → usage spiked → cost went up. Usage Metering tells you when and what; Audit Trail tells you who made the change.
Prerequisites
pup auth login # OAuth2 (recommended) — covers audit queries
# Usage Metering queries also need DD_API_KEY + DD_APP_KEY
export DD_API_KEY=<your-api-key>
export DD_APP_KEY=<your-app-key>
export DD_SITE=datadoghq.com
Scope Boundary
This skill identifies configuration changes that may have caused a spike. It does not identify which specific user or process submitted the data (e.g., which service sent the LLM spans). For per-submission attribution, use LLM Observability traces or APM instrumentation.
Investigation Workflow
Step 1 — Identify the spike window and product family
START=$(date -u -v-7d +"%Y-%m-%dT%H:%M:%SZ" 2>/dev/null || date -u -d "7 days ago" +"%Y-%m-%dT%H:%M:%SZ")
END=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
curl -s -G "https://api.${DD_SITE}/api/v2/usage/hourly_usage" \
-H "DD-API-KEY: ${DD_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DD_APP_KEY}" \
--data-urlencode "filter[timestamp][start]=${START}" \
--data-urlencode "filter[timestamp][end]=${END}" \
--data-urlencode "filter[product_families]=all" \
| jq '[.data[] | {
timestamp: .attributes.timestamp,
product: .attributes.product_family,
measurements: [.attributes.measurements[] | {type: .usage_type, value: .value}]
}]'
Product families with LLM/AI coverage: llm_observability, bits_ai, logs, apm
Step 2 — Pinpoint the spike
From Step 1, identify the hour/day where volume jumped. Note the timestamp as SPIKE_TIME.
Step 3 — Search Audit Trail for config changes in the 24h preceding the spike
pup audit-logs search \
--query "@action:(created OR modified OR deleted)" \
--from "SPIKE_TIME_MINUS_24H" \
--to "SPIKE_TIME" \
--limit 200 \
-o json \
| jq '[.data[] | {
timestamp: .attributes.timestamp,
user: .attributes.attributes.usr.email,
actor_type: .attributes.attributes.evt.actor.type,
action: .attributes.attributes.action,
event_category: .attributes.attributes.evt.name,
resource_type: .attributes.attributes.asset.type,
resource_id: .attributes.attributes.asset.id
}]'
Note:
--fromand--toaccept ISO timestamps (e.g.,2026-05-01T14:00:00Z) or relative values (1h,24h,7d).
Step 4 — Narrow to product-relevant config changes
Filter to the audit categories most likely to affect the spiking product:
| If this product spiked | Add to query |
|---|---|
llm_observability | @evt.name:(Integration OR APM OR "Log Management") |
logs / indexed_logs | @evt.name:"Log Management" @asset.type:(pipeline OR index OR exclusion_filter) |
apm / indexed_spans | @evt.name:APM @asset.type:(retention_filter OR sampling_rate) |
rum | @evt.name:RUM |
metrics | @evt.name:Metrics |
Example for LLM Observability spike:
pup audit-logs search \
--query "@evt.name:(Integration OR APM OR \"Log Management\") @action:(created OR modified)" \
--from "SPIKE_TIME_MINUS_24H" \
--to "SPIKE_TIME" \
--limit 100 \
-o json \
| jq '[.data[] | {
timestamp: .attributes.timestamp,
user: .attributes.attributes.usr.email,
action: .attributes.attributes.action,
category: .attributes.attributes.evt.name,
resource_type: .attributes.attributes.asset.type,
resource_id: .attributes.attributes.asset.id
}]'
Output Format
Usage spike detected:
Product: <product_family>
Spike time: <SPIKE_TIME>
Volume: <baseline> → <spike_value> (<magnitude>×)
Configuration changes in 24h preceding spike:
<timestamp> | <user_email> | <action> <resource_type> <resource_id> | <category>
Likely causal change: <most-proximate change matching the product family>
Confidence: HIGH (single clear change) / MEDIUM (multiple candidates) / LOW (no matching changes)
Next steps:
- Confirm with <user_email> whether the change was intentional
- If unintentional: revert <resource_id> and monitor volume
- If intentional: update cost forecasts and alert thresholds
When No Causal Change Is Found
- The change may predate the 24h window — expand to 72h
- The increase may be from application-side instrumentation changes — check deploys
- The increase may be organic traffic growth — correlate with product launch or traffic event
References
FAQ
What does the dd-audit-cost-spike-investigation skill do?
Investigate a Datadog product usage or cost spike by correlating Usage Metering data (when/what spiked) with Audit Trail config changes (who changed what in the preceding window).
How do I install dd-audit-cost-spike-investigation?
Run npx skills add https://github.com/datadog-labs/agent-skills --skill dd-audit-cost-spike-investigation in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/dd-audit-cost-spike-investigation/ (personal) or .claude/skills/dd-audit-cost-spike-investigation/ (project).
Which AI agents can use dd-audit-cost-spike-investigation?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is dd-audit-cost-spike-investigation safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains dd-audit-cost-spike-investigation?
It is published by datadog-labs in datadog-labs/agent-skills, and it is an official skill from the maker of the technology it covers. It has 476 recorded installs.
More from datadog-labs/agent-skills
dd-pup
datadog-labs/agent-skills
Datadog CLI (Rust). OAuth2 auth with token refresh.
agent-skills
datadog-labs/agent-skills
Datadog skills for AI agents. Essential monitoring, logging, tracing and observability.
dd-logs
datadog-labs/agent-skills
Log management - search, archives, metrics, and cost control.
dd-apm
datadog-labs/agent-skills
APM - install, onboard, instrument, enable, set up, configure, traces, services, dependencies, performance analysis, Data Streams Monitoring (DSM), queue lag, pipeline latency. Use for any request involving Datadog APM setup, instrumentation (SSI, ddtrace, agent install), or analysis.
dd-monitors
datadog-labs/agent-skills
Monitor management - list, search, file-based create, and alerting best practices.
dd-docs
datadog-labs/agent-skills
Datadog docs lookup using docs.datadoghq.com/llms.txt and linked Markdown pages.
Popular security skills
azure-compliance
microsoft/azure-skills
Run Azure compliance and security audits with azqr plus Key Vault expiration checks. Covers best-practice assessment, resource review, policy/compliance validation, and security posture checks. WHEN: compliance scan, security audit, BEFORE running azqr (compliance cli tool), Azure best practices, Key Vault expiration check, expired certificates, expiring secrets, orphaned resources, compliance assessment.
minimal-run-and-audit
lllllllama/rigorpilot-skills
Rigor Run skill for README-first deep learning repo reproduction. Use when the task is specifically to capture or normalize evidence from the selected smoke test or documented inference or evaluation command and write standardized reprooutputs/ files, including patch notes when repository files changed. Do not use for training execution, initial repo intake, generic environment setup, paper lookup, target selection, hidden scientific-meaning changes, or end-to-end orchestration by itself.
seo-audit
coreyhaines31/marketingskills
When the user wants to audit, review, or diagnose SEO issues on their site. Also use when the user mentions "SEO audit," "technical SEO," "why am I not ranking," "SEO issues," "on-page SEO," "meta tags review," "SEO health check," "my traffic dropped," "lost rankings," "not showing up in Google," "site isn't ranking," "Google update hit me," "page speed," "core web vitals," "crawl errors," "indexing issues," "stuck on page 2," "striking distance keywords," "title tag rewrite," "low CTR," "local SEO," "Google Business Profile," or "map pack." Use this even if the user just says something vague like "my SEO is bad" or "help with SEO" — start with an audit. For building pages at scale to target keywords, see programmatic-seo. For adding structured data, see schema. For AI search optimization, see ai-seo.
firebase-security-rules-auditor
firebase/agent-skills
Audits Firebase (Firestore, Cloud Storage) security rules for vulnerabilities, privilege escalation, role bypasses, create vs update inconsistencies, resource exhaustion, type safety, size limits, and hasOnly ownership checks. Use when auditing/reviewing rules, running red-team rule assessments, or scoring against auditor checklists. Don't use for Firebase CLI (login, deploy), Auth, Crashlytics, Remote Config, or database queries.
audit
pbakaus/impeccable
Run technical quality checks across accessibility, performance, theming, responsive design, and anti-patterns. Generates a scored report with P0-P3 severity ratings and actionable plan. Use when the user wants an accessibility check, performance audit, or technical quality review.
competitors
coreyhaines31/marketingskills
When the user wants to create competitor comparison or alternative pages for SEO and buyer-facing use. Also use when the user mentions 'alternative page,' 'vs page,' 'competitor comparison,' 'comparison page,' '[Product] vs [Product],' '[Product] alternative,' 'competitive landing pages,' 'how do we compare to X,' 'competitor teardown,' 'audit our competitor pages,' 'are our comparison pages out of date,' or 'competitive asset audit.' Use this for any content that positions your product against competitors. Covers four formats: singular alternative, plural alternatives, you vs competitor, and competitor vs competitor. For auditing existing claims (not researching competitors from scratch, which is competitor-profiling; not technical SEO on these pages, which is seo-audit), use the asset audit here. For internal battle cards and sales-specific competitor docs, see sales-enablement.