sentry-setup-ai-monitoring
Setup Sentry AI Agent Monitoring in any project. Use when asked to monitor LLM calls, track AI agents, or instrument OpenAI/Anthropic/Vercel AI/LangChain/Google GenAI/Pydantic AI. Detects installed AI SDKs and configures appropriate integrations.
- Installs
- 894
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/getsentry/sentry-agent-skills --skill sentry-setup-ai-monitoring- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads sentry-setup-ai-monitoring automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Setup Sentry AI Agent Monitoring
Configure Sentry to track LLM calls, agent executions, tool usage, and token consumption.
Invoke This Skill When
- User asks to "monitor AI/LLM calls" or "track OpenAI/Anthropic usage"
- User wants "AI observability" or "agent monitoring"
- User asks about token usage, model latency, or AI costs
Important: The SDK versions, API names, and code samples below are examples. Always verify against docs.sentry.io before implementing, as APIs and minimum versions may have changed.
Prerequisites
AI monitoring requires tracing enabled (tracesSampleRate > 0).
Data Capture Warning
Prompt and output recording captures user content that is likely PII. Before enabling recordInputs/recordOutputs (JS) or include_prompts/send_default_pii (Python), confirm:
- The application's privacy policy permits capturing user prompts and model responses
- Captured data complies with applicable regulations (GDPR, CCPA, etc.)
- Sentry data retention settings are appropriate for the sensitivity of the data
Ask the user whether they want prompt/output capture enabled. Do not enable it by default — configure it only when explicitly requested or confirmed. Use tracesSampleRate: 1.0 only in development; in production, use a lower value or a tracesSampler function.
Detection First
Always detect installed AI SDKs before configuring:
# JavaScript
grep -E '"(openai|@anthropic-ai/sdk|ai|@langchain|@google/genai)"' package.json
# Python
grep -E '(openai|anthropic|langchain|huggingface)' requirements.txt pyproject.toml 2>/dev/null
Supported SDKs
JavaScript
| Package | Integration | Min Sentry SDK | Auto? |
|---|---|---|---|
openai | openAIIntegration() | 10.28.0 | Yes |
@anthropic-ai/sdk | anthropicAIIntegration() | 10.28.0 | Yes |
ai (Vercel) | vercelAIIntegration() | 10.6.0 | Yes* |
@langchain/* | langChainIntegration() | 10.28.0 | Yes |
@langchain/langgraph | langGraphIntegration() | 10.28.0 | Yes |
@google/genai | googleGenAIIntegration() | 10.28.0 | Yes |
*Vercel AI: 10.6.0+ for Node.js, Cloudflare Workers, Vercel Edge Functions, Bun. 10.12.0+ for Deno. Requires experimental_telemetry per-call.
Python
Integrations auto-enable when the AI package is installed — no explicit registration needed:
| Package | Auto? | Notes |
|---|---|---|
openai | Yes | Includes OpenAI Agents SDK |
anthropic | Yes | |
langchain / langgraph | Yes | |
huggingface_hub | Yes | |
google-genai | Yes | |
pydantic-ai | Yes | |
litellm | No | Requires explicit integration |
mcp (Model Context Protocol) | Yes |
JavaScript Configuration
Node.js — auto-enabled integrations
Just ensure tracing is enabled. Integrations auto-enable when the AI package is installed:
Sentry.init({
dsn: "YOUR_DSN",
tracesSampleRate: 1.0, // Lower in production (e.g., 0.1)
// OpenAI, Anthropic, Google GenAI, LangChain integrations auto-enable in Node.js
});
To customize (e.g., enable prompt capture — see Data Capture Warning):
integrations: [
Sentry.openAIIntegration({
// recordInputs: true, // Opt-in: captures prompt content (PII)
// recordOutputs: true, // Opt-in: captures response content (PII)
}),
],
Browser / Next.js OpenAI (manual wrapping required)
In browser-side code or Next.js meta-framework apps, auto-instrumentation is not available. Wrap the client manually:
import OpenAI from "openai";
import * as Sentry from "@sentry/nextjs"; // or @sentry/react, @sentry/browser
const openai = Sentry.instrumentOpenAiClient(new OpenAI());
// Use 'openai' client as normal
LangChain / LangGraph (auto-enabled)
integrations: [
Sentry.langChainIntegration({
// recordInputs: true, // Opt-in: captures prompt content (PII)
// recordOutputs: true, // Opt-in: captures response content (PII)
}),
Sentry.langGraphIntegration({
// recordInputs: true,
// recordOutputs: true,
}),
],
Vercel AI SDK
Add to sentry.edge.config.ts for Edge runtime:
integrations: [Sentry.vercelAIIntegration()],
Enable telemetry per-call:
await generateText({
model: openai("gpt-4o"),
prompt: "Hello",
experimental_telemetry: {
isEnabled: true,
// recordInputs: true, // Opt-in: captures prompt content (PII)
// recordOutputs: true, // Opt-in: captures response content (PII)
},
});
Python Configuration
Integrations auto-enable — just init with tracing. Only add explicit imports to customize options:
import sentry_sdk
sentry_sdk.init(
dsn="YOUR_DSN",
traces_sample_rate=1.0, # Lower in production (e.g., 0.1)
# send_default_pii=True, # Opt-in: required for prompt capture (sends user PII)
# Integrations auto-enable when the AI package is installed.
# Only specify explicitly to customize (e.g., include_prompts):
# integrations=[OpenAIIntegration(include_prompts=True)],
)
Manual Instrumentation
Use when no supported SDK is detected.
Span Types
op Value | Purpose |
|---|---|
gen_ai.request | Individual LLM calls |
gen_ai.invoke_agent | Agent execution lifecycle |
gen_ai.execute_tool | Tool/function calls |
gen_ai.handoff | Agent-to-agent transitions |
Example (JavaScript)
await Sentry.startSpan({
op: "gen_ai.request",
name: "LLM request gpt-4o",
attributes: { "gen_ai.request.model": "gpt-4o" },
}, async (span) => {
span.setAttribute("gen_ai.request.messages", JSON.stringify(messages));
const result = await llmClient.complete(prompt);
span.setAttribute("gen_ai.usage.input_tokens", result.inputTokens);
span.setAttribute("gen_ai.usage.output_tokens", result.outputTokens);
return result;
});
Key Attributes
| Attribute | Description |
|---|---|
gen_ai.request.model | Model identifier |
gen_ai.request.messages | JSON input messages |
gen_ai.usage.input_tokens | Input token count |
gen_ai.usage.output_tokens | Output token count |
gen_ai.agent.name | Agent identifier |
gen_ai.tool.name | Tool identifier |
Enable prompt/output capture only after confirming with the user (see Data Capture Warning above).
Verification
After configuring, make an LLM call and check the Sentry Traces dashboard. AI spans appear with gen_ai.* operations showing model, token counts, and latency.
Troubleshooting
| Issue | Solution |
|---|---|
| AI spans not appearing | Verify tracesSampleRate > 0, check SDK version |
| Token counts missing | Some providers don't return tokens for streaming |
| Prompts not captured | Enable recordInputs/include_prompts |
| Vercel AI not working | Add experimental_telemetry to each call |
FAQ
What does the sentry-setup-ai-monitoring skill do?
Setup Sentry AI Agent Monitoring in any project. Use when asked to monitor LLM calls, track AI agents, or instrument OpenAI/Anthropic/Vercel AI/LangChain/Google GenAI/Pydantic AI. Detects installed AI SDKs and configures appropriate integrations.
How do I install sentry-setup-ai-monitoring?
Run npx skills add https://github.com/getsentry/sentry-agent-skills --skill sentry-setup-ai-monitoring in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/sentry-setup-ai-monitoring/ (personal) or .claude/skills/sentry-setup-ai-monitoring/ (project).
Which AI agents can use sentry-setup-ai-monitoring?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is sentry-setup-ai-monitoring safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains sentry-setup-ai-monitoring?
It is published by getsentry in getsentry/sentry-agent-skills, and it is an official skill from the maker of the technology it covers. It has 894 recorded installs.
More from getsentry/sentry-agent-skills
sentry-fix-issues
getsentry/sentry-agent-skills
Find and fix issues from Sentry using MCP. Use when asked to fix Sentry errors, debug production issues, investigate exceptions, or resolve bugs reported in Sentry. Methodically analyzes stack traces, breadcrumbs, traces, and context to identify root causes.
sentry-react-setup
getsentry/sentry-agent-skills
Setup Sentry in React apps. Use when asked to add Sentry to React, install @sentry/react, or configure error monitoring, error boundaries, session replay, or browser tracing for React applications.
sentry-nextjs-sdk
getsentry/sentry-agent-skills
Full Sentry SDK setup for Next.js. Use when asked to "add Sentry to Next.js", "install @sentry/nextjs", or configure error monitoring, tracing, session replay, logging, profiling, AI monitoring, or crons for Next.js applications. Supports Next.js 13+ with App Router and Pages Router.
sentry-pr-code-review
getsentry/sentry-agent-skills
Review a project's PRs to check for issues detected in code review by Seer Bug Prediction. Use when asked to review or fix issues identified by Sentry in PR comments, or to find recent PRs with Sentry feedback.
sentry-create-alert
getsentry/sentry-agent-skills
Create Sentry alerts using the workflow engine API. Use when asked to create alerts, set up notifications, configure issue priority alerts, or build workflow automations. Supports email, Slack, PagerDuty, Discord, and other notification actions.
sentry-react-sdk
getsentry/sentry-agent-skills
Full Sentry SDK setup for React. Use when asked to "add Sentry to React", "install @sentry/react", or configure error monitoring, tracing, session replay, profiling, or logging for React applications. Supports React 16+, React Router v5-v7, TanStack Router, Redux, Vite, and webpack.
Popular devops & cloud skills
microsoft-foundry
microsoft/azure-skills
Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent create, managed harness agent create, GitHub Copilot harness agent, create agent, update agent, add tool to agent, invoke agent, agent insights, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure
azure-diagnostics
microsoft/azure-skills
Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.
azure-prepare
microsoft/azure-skills
Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.
azure-ai
microsoft/azure-skills
Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.
azure-deploy
microsoft/azure-skills
Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/deployment-plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.
azure-validate
microsoft/azure-skills
Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity permissions, and prerequisites before deploying. WHEN: validate my app, check deployment readiness, run preflight checks, verify configuration, check if ready to deploy, validate azure.yaml, validate Bicep, test before deploying, troubleshoot deployment errors, validate Azure Functions, validate function app, validate serverless deployment, verify RBAC roles, check role assignments, review managed identity permissions, what-if analysis, validate Container Apps deployment.