google-agents-cli-observability
This skill should be used when the user wants to "set up tracing", "monitor my agent", "configure logging", "add observability", "debug production traffic", or needs guidance on monitoring deployed agents, including ADK (Agent Development Kit) agents. Covers Cloud Trace, prompt-response logging, BigQuery Agent Analytics, third-party integrations (AgentOps, Phoenix, MLflow, etc.), and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment setup (use google-agents-cli-deploy) or API code patterns (use google-agents-cli-adk-code).
- Installs
- 513.4K
- Last 24h
- +2K
- Trending rank
- #174
- Security
- Pass
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/google/agents-cli --skill google-agents-cli-observability- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads google-agents-cli-observability automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
Security audits
Independent scans of the skill's instructions and bundled files.
SKILL.md
The instructions your agent reads when the skill is triggered.
Observability Guide
Cloud Trace works out of the box — no infrastructure needed. Prompt-response logging and BigQuery Agent Analytics require Terraform-provisioned infrastructure (service account, GCS bucket, BigQuery dataset). Run
agents-cli infra single-project --apply --project PROJECT_IDto provision these resources. Go projects get the BigQuery telemetry stack too; the GCS completion upload behind prompt-response logging and the BigQuery Agent Analytics plugin are Python only. Seereferences/cloud-trace-and-logging.mdfor details, env vars, and verification commands. If your project isn't scaffolded yet, see/google-agents-cli-scaffoldfirst.
Deployment order
| Do you want Terraform-managed observability? | Action |
|---|---|
| No | Run agents-cli deploy directly (works for all targets). |
| Yes | Run agents-cli infra single-project --apply first, then agents-cli deploy. |
If you already ran agents-cli deploy imperatively:
Do not apply Terraform afterward. For Agent Runtime in Gemini platform, Terraform owns the whole engine (service account, deployment spec, env vars), so a later apply can't be reconciled without taking over the resource.
Either delete the deployment and start over, or keep the SDK-deployed instance — skip infra single-project and set the observability env vars by re-running agents-cli deploy --update-env-vars "KEY=VALUE,..."; deploy matches the existing Reasoning Engine by display name and updates it in place, preserving env vars set outside the deploy. You must also grant its service account the telemetry IAM roles the Terraform module would otherwise provision: roles/storage.admin (write completions to the logs bucket), roles/logging.logWriter, roles/cloudtrace.agent, plus roles/bigquery.dataOwner + roles/bigquery.jobUser when scaffolded with --bq-analytics. The full set lives in deployment/terraform/single-project/iam.tf (from app_sa_roles) and telemetry.tf. Terraform-managed env vars aren't available in this mode.
Reference Files
| File | Contents |
|---|---|
references/cloud-trace-and-logging.md | Scaffolded project details — Terraform-provisioned resources, environment variables, verification commands, enabling/disabling locally |
references/bigquery-agent-analytics.md | BQ Agent Analytics plugin — enabling, key features, GCS offloading, tool provenance |
references/adk-docs.md | ADK: adk.dev pages to fetch for detail beyond this skill |
references/feedback-mechanism.md | Adding a user-feedback endpoint — request model, structured logging, log sink → BigQuery |
Observability Tiers
Choose the right level of observability based on your needs:
| Tier | What It Does | Scope | Default State | Best For |
|---|---|---|---|---|
| Cloud Trace | Distributed tracing — execution flow, latency, errors via OpenTelemetry spans | All templates, all environments | Always enabled | Debugging latency, understanding agent execution flow |
| Prompt-Response Logging | GenAI interactions exported to GCS, BigQuery, and Cloud Logging | Scaffolded ADK Python projects | Disabled locally, enabled when deployed | Auditing LLM interactions, compliance |
| BigQuery Agent Analytics | Structured agent events (LLM calls, tool use, outcomes) to BigQuery | ADK Python agents with the plugin enabled | Opt-in (--bq-analytics at scaffold time) | Conversational analytics, custom dashboards, LLM-as-judge evals |
| Third-Party Integrations | External observability platforms (AgentOps, Phoenix, MLflow, etc.) | Any OpenTelemetry-instrumented agent | Opt-in, per-provider setup | Team collaboration, specialized visualization, prompt management |
Ask the user which tier(s) they need — they can be combined. Cloud Trace is always on; the others are additive.
Cloud Trace
Scaffolded agents use OpenTelemetry to emit distributed traces. Every agent invocation produces spans that track the full execution flow.
Span Hierarchy
ADK projects. These are ADK's span names; other frameworks emit their own (
generate_contentcomes from the shared google-genai instrumentor either way).
invoke_workflow (top-level run)
└── invoke_agent (one per agent in the chain)
├── call_llm (model request)
│ └── generate_content (underlying GenAI model call)
└── execute_tool (tool execution)
Setup by Deployment Type
| Deployment | Setup |
|---|---|
| Agent Runtime | Automatic — exporters wired at startup, gated on GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY (set by deploy); exports to Cloud Trace/Logging + the Agent Runtime page in the Google Cloud console |
| Cloud Run / GKE (scaffolded) | Automatic — exporters wired at startup, exports to Cloud Trace/Logging |
| Cloud Run / GKE (manual) | Configure OpenTelemetry exporter in your app |
| Local dev | Works with agents-cli playground; traces visible in Cloud Console |
Wired at app startup — ADK Python: get_fast_api_app(otel_to_cloud=True) in app/fast_api_app.py; ADK Go: setupObservability() in observability.go; other templates call their own. references/cloud-trace-and-logging.md has the details.
View traces: Cloud Console → Trace → Trace explorer
ADK: for detailed setup instructions (Agent Runtime CLI/SDK, Cloud Run, custom deployments), fetch https://adk.dev/integrations/cloud-trace/index.md.
Prompt-Response Logging
Captures GenAI interactions and exports to GCS (JSONL) and BigQuery (via log sinks + external tables). Content is governed by two independent tiers; the net Terraform-deploy default is full content in GCS/BigQuery, none in traces:
| Tier | Captures | Controlled by | Default (Terraform deploy) |
|---|---|---|---|
| GCS/BigQuery completions | Full prompts/responses (the prompt-response logging feature) | OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK=upload + LOGS_BUCKET_NAME | On — full content |
| Trace spans / Cloud Logging events | Span/event content | OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT (plus ADK_CAPTURE_MESSAGE_CONTENT_IN_SPANS=false, ADK Python only) | Off — NO_CONTENT |
The tiers are independent: GCS/BigQuery uploads capture full content whenever their upload vars are set and do not honor OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, which governs the traces/events tier only.
ADK Python reads it as the experimental-semconv enum:
NO_CONTENT— no content in spans/events (scaffolded default)EVENT_ONLY— content in Cloud Logging eventsSPAN_ONLY/SPAN_AND_EVENT— content in trace spanstrue/false— invalid; fall back toNO_CONTENT
ADK Go reads the same variable as a boolean: "1" or "true" capture content, every other
value — including the enum members above — elides it.
For the full mechanics (semconv opt-in, declarative Terraform config, env-var table, enabling/disabling, verification commands), see references/cloud-trace-and-logging.md. For ADK logging docs (log levels, configuration, debugging), fetch https://adk.dev/observability/logging/index.md.
BigQuery Agent Analytics Plugin
ADK projects. Optional ADK plugin that logs structured agent events to BigQuery. Enable with
--bq-analyticsat scaffold time. Seereferences/bigquery-agent-analytics.mdfor details.
Third-Party Integrations
Many third-party observability platforms can ingest agent telemetry (via OpenTelemetry or custom instrumentation). The table below covers common ones; the full list is larger (see the pointer below it).
| Platform | Key Differentiator | Setup Complexity | Self-Hosted Option |
|---|---|---|---|
| AgentOps | Session replays, 2-line setup, replaces native telemetry | Minimal | No (SaaS) |
| Arize AX | Commercial platform, production monitoring, evaluation dashboards | Low | No (SaaS) |
| Phoenix | Open-source, custom evaluators, experiment testing | Low | Yes |
| MLflow | OTel traces to MLflow Tracking Server, span tree visualization | Medium (needs SQL backend) | Yes |
| Monocle | 1-call setup, VS Code Gantt chart visualizer | Minimal | Yes (local files) |
| Weave | W&B platform, team collaboration, timeline views | Low | No (SaaS) |
| Freeplay | Prompt management + evals + observability in one platform | Low | No (SaaS) |
Ask the user which platform they prefer — present the trade-offs and let them choose. ADK: fetch a platform's setup page at https://adk.dev/integrations/<slug>/index.md (slugs for the table above: agentops, arize-ax, phoenix, mlflow-tracing, monocle, weave, freeplay); ADK has more observability integrations (Datadog, Galileo, LangWatch, Latitude, Future AGI, Respan, Zespan, …) — browse the complete, current list at https://adk.dev/integrations/ (observability topic). On other frameworks the OpenTelemetry-based platforms still work, but follow the platform's own setup docs.
Troubleshooting
| Issue | Solution |
|---|---|
| No traces in Cloud Trace | Verify telemetry setup runs at startup and the SA has the cloudtrace.agent role. ADK Python: fast_api_app.py uses get_fast_api_app(otel_to_cloud=True); ADK Go: observability.go build the exporter manually. Agent Runtime additionally gates this on GOOGLE_CLOUD_AGENT_ENGINE_ENABLE_TELEMETRY. |
| Prompt-response data not appearing | Check LOGS_BUCKET_NAME is set; verify SA has storage.objectCreator on the bucket; check app logs for telemetry setup warnings |
| Content in traces/events (unwanted) | ADK Python: OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=NO_CONTENT keeps content out of spans/events. ADK Go: any value other than 1/true does, so unset it or set false. NOTE: GCS/BigQuery completions still capture full content — to stop that, remove LOGS_BUCKET_NAME/OTEL_INSTRUMENTATION_GENAI_COMPLETION_HOOK (drop the upload block in service.tf) |
| BigQuery Analytics not logging | ADK Python: verify the plugin is configured in app/agent.py; check BQ_ANALYTICS_DATASET_ID env var is set |
| Third-party integration not capturing spans | Check provider-specific env vars (API keys, endpoints); some providers (AgentOps) replace native telemetry |
| Traces missing tool spans | ADK: tool execution spans appear under execute_tool (other frameworks use their own span names) — check trace explorer filters |
| High telemetry costs | Turn content capture off (NO_CONTENT in Python, false in Go); reduce BigQuery retention; disable unused tiers |
Related Skills
/google-agents-cli-deploy— Deployment targets, CI/CD pipelines, and production workflows/google-agents-cli-workflow— Development workflow, coding guidelines, and operational rules/google-agents-cli-adk-code— ADK API quick reference for writing agent code, Python and Go
FAQ
What does the google-agents-cli-observability skill do?
This skill should be used when the user wants to "set up tracing", "monitor my agent", "configure logging", "add observability", "debug production traffic", or needs guidance on monitoring deployed agents, including ADK (Agent Development Kit) agents. Covers Cloud Trace, prompt-response logging, BigQuery Agent Analytics, third-party integrations (AgentOps, Phoenix, MLflow, etc.), and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment setup (use google-agents-cli-deploy) or API code patterns (use google-agents-cli-adk-code).
How do I install google-agents-cli-observability?
Run npx skills add https://github.com/google/agents-cli --skill google-agents-cli-observability in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/google-agents-cli-observability/ (personal) or .claude/skills/google-agents-cli-observability/ (project).
Which AI agents can use google-agents-cli-observability?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is google-agents-cli-observability safe to install?
Security partners have reviewed it: Gen Agent Trust Hub passed; Socket passed; Snyk passed. As with any skill, read the SKILL.md and bundled scripts before installing.
Who maintains google-agents-cli-observability?
It is published by google in google/agents-cli. It has 513,353 recorded installs.
More from google/agents-cli
google-agents-cli-adk-code
google/agents-cli
This skill should be used when the user wants to "write agent code", "build an agent with ADK", "add a tool", "create a callback", "define an agent", "use state management" — in a project that needs ADK (Agent Development Kit) API patterns and code examples. It provides a quick reference for agent types, tool definitions, orchestration patterns, callbacks, state management, the graph Workflow API, and reference recipes to study. Do NOT use for scaffolding (use google-agents-cli-scaffold) or deployment (use google-agents-cli-deploy).
google-agents-cli-deploy
google/agents-cli
This skill should be used when the user wants to "deploy an agent", "deploy my ADK agent", "set up CI/CD", "configure secrets", "troubleshoot a deployment", or needs guidance on Agent Runtime, Cloud Run, or GKE deployment targets, or binding an agent to an Agent Gateway. Covers deployment workflows, service accounts, rollback, and production infrastructure. Applies to any framework agents-cli deploys (ADK, LangChain, ...). Part of the agents-cli skills suite. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), evaluation (use google-agents-cli-eval), or project scaffolding (use google-agents-cli-scaffold).
google-agents-cli-publish
google/agents-cli
This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).
google-agents-cli-workflow
google/agents-cli
This skill should be used when the user wants to "develop an agent", "build an agent using ADK", "run the agent locally", "debug agent code", "test an agent", "deploy an agent", "publish an agent", "monitor an agent", or needs the ADK (Agent Development Kit) development lifecycle and coding guidelines. Entrypoint for building ADK agents. Always active — provides the full workflow (scaffold, build, evaluate, deploy, publish, observe), code preservation rules, model selection guidance, and troubleshooting steps for ADK or any agent development.
google-agents-cli-eval
google/agents-cli
This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
google-agents-cli-scaffold
google/agents-cli
This skill should be used when the user wants to "create an agent project", "start a new ADK project", "build me a new agent", "add CI/CD to my project", "add deployment", "enhance my project", or "upgrade my project". Part of the agents-cli skills suite. Covers agents-cli scaffold create, scaffold enhance, and scaffold upgrade commands, template options, deployment targets, and the prototype-first workflow. Do NOT use for writing agent code (ADK projects: use google-agents-cli-adk-code) or deployment operations (use google-agents-cli-deploy).
Popular devops & cloud skills
microsoft-foundry
microsoft/azure-skills
Build, deploy, evaluate, optimize, fine-tune, and manage Microsoft Foundry agents, models, and resources end to end. USE FOR: foundry, azd ai agent, azd provision/deploy, hosted agent scaffold/develop/run/deploy/troubleshoot, prompt agent create, managed harness agent create, GitHub Copilot harness agent, create agent, update agent, add tool to agent, invoke agent, agent insights, evaluate agent, batch eval, continuous eval, continuous monitoring, agent CI/CD, optimize prompt, improve prompt, optimize agent instructions, Agent Optimizer scaffold, dataset curation from traces, deploy model, model fine-tuning (SFT/DPO/RFT), Foundry project, RBAC, role assignment, permissions, quota, capacity, region, deployment failure, AI Services, create Foundry resource, knowledge index, customize deployment, onboard, availability, training-data, grader, distillation, large file upload. DO NOT USE FOR: Azure Functions, App Service, general Azure deploy (use azure-deploy), general Azure prep (use azure
azure-diagnostics
microsoft/azure-skills
Debug Azure production issues on Azure using AppLens, Azure Monitor, resource health, and safe triage. WHEN: debug production issues, troubleshoot app service, app service high CPU, app service deployment failure, troubleshoot container apps, container app 503 or target port mismatch, troubleshoot functions, function telemetry missing or stopped after a config/connection-string change, troubleshoot AKS, VM RDP, Linux SSH, VM black screen, can't connect to VM, reset VM password, NSG or firewall blocking, kubectl cannot connect, kube-system/CoreDNS failures, pod pending, crashloop, node not ready, upgrade failures, analyze logs, KQL, insights, image pull failures, cold start issues, health probe failures, resource health, root cause of errors, troubleshoot event hubs, troubleshoot service bus, messaging SDK error, AMQP connection failure, message lock lost, lock renewal during long-running processing, service bus dead letter.
azure-prepare
microsoft/azure-skills
Prepare azd-based Azure projects for deployment: generates azure.yaml, infrastructure (Bicep/Terraform), and Dockerfiles for the Azure Developer CLI (azd) workflow. USE ONLY when the user explicitly wants to use azd as the deployment tool, or the project already has an azure.yaml file. DO NOT USE FOR: non-azd deployments, Python App Service code-only deploys (use python-appservice-deploy), or cross-cloud migration (use azure-cloud-migrate). WHEN: prepare app for azd, create azure.yaml, set up azd infrastructure, modernize app for Azure with azd, deploy with azd, function app, timer trigger, service bus trigger, event-driven function, managed identity, generate Bicep, generate Terraform, create and deploy to Azure.
azure-ai
microsoft/azure-skills
Use for Azure AI: Search, Speech, OpenAI, Document Intelligence. Helps with search, vector/hybrid search, speech-to-text, text-to-speech, transcription, OCR. WHEN: AI Search, query search, vector search, hybrid search, semantic search, speech-to-text, text-to-speech, transcribe, OCR, convert text to speech.
azure-deploy
microsoft/azure-skills
Execute Azure deployments for ALREADY-PREPARED applications that have existing .azure/deployment-plan.md and infrastructure files. DO NOT use this skill when the user asks to CREATE a new application — use azure-prepare instead. This skill runs azd up, azd deploy, terraform apply, and az deployment commands with built-in error recovery. Requires .azure/deployment-plan.md from azure-prepare and validated status from azure-validate. WHEN: "run azd up", "run azd deploy", "execute deployment", "push to production", "push to cloud", "go live", "ship it", "bicep deploy", "terraform apply", "publish to Azure", "launch on Azure". DO NOT USE WHEN: "create and deploy", "build and deploy", "create a new app", "set up infrastructure", "create and deploy to Azure using Terraform" — use azure-prepare for these.
azure-validate
microsoft/azure-skills
Pre-deployment validation for Azure readiness. Run deep checks on configuration, infrastructure (Bicep or Terraform), RBAC role assignments, managed identity permissions, and prerequisites before deploying. WHEN: validate my app, check deployment readiness, run preflight checks, verify configuration, check if ready to deploy, validate azure.yaml, validate Bicep, test before deploying, troubleshoot deployment errors, validate Azure Functions, validate function app, validate serverless deployment, verify RBAC roles, check role assignments, review managed identity permissions, what-if analysis, validate Container Apps deployment.