instrument-llm-analytics
Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.
- Installs
- 262
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/posthog/ai-plugin --skill instrument-llm-analytics- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads instrument-llm-analytics automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Add PostHog AI Observability
Use this skill to add PostHog AI Observability that traces AI model usage in new or changed code. Use it after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. If PostHog is not yet installed, this skill also covers initial SDK setup. Supports any provider or framework.
Supported providers: OpenAI, Azure OpenAI, Anthropic, Google, Cohere, Mistral, Perplexity, DeepSeek, Groq, Together AI, Fireworks AI, xAI, Cerebras, Hugging Face, Ollama, OpenRouter.
Supported frameworks: LangChain, LlamaIndex, CrewAI, AutoGen, DSPy, LangGraph, Pydantic AI, Vercel AI, LiteLLM, Instructor, Semantic Kernel, Mirascope, Mastra, SmolAgents, OpenAI Agents.
Proxy/gateway: Portkey, Helicone.
Instructions
Follow these steps IN ORDER:
STEP 1: Analyze the codebase and detect the LLM stack.
- Look for LLM provider SDKs (openai, anthropic, google-generativeai, etc.) and AI frameworks (langchain, llamaindex, crewai, etc.) in dependency files and imports.
- Look for lockfiles to determine the package manager.
- Check for existing PostHog or observability setup. If PostHog is already installed and LLM tracing is configured, skip to STEP 4 to add tracing for any new LLM calls.
STEP 2: Research instrumentation. (Skip if PostHog LLM tracing is already set up.) 2.1. Find the reference file below that matches the detected provider or framework — it is the source of truth for callback setup, middleware configuration, and event capture. Read it now. 2.2. If no reference matches, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.
STEP 3: Install the PostHog SDK. (Skip if PostHog is already set up.)
- Add the PostHog SDK and any required callback/integration packages.
- Do not manually edit dependency files — use the package manager's install command.
STEP 4: Add LLM tracing.
- Instrument LLM calls to capture input tokens, output tokens, model name, latency, and costs for every generation.
- Follow the provider-specific reference for the exact callback/middleware setup.
- Do not alter the fundamental architecture of existing files. Make additions minimal and targeted.
- You must read a file immediately before attempting to write it.
STEP 5: Link to users.
- Associate LLM generations with identified users via distinct IDs when possible.
STEP 6: Set up environment variables.
- Check if the project already has PostHog environment variables configured (e.g. in
.env,.env.local, or framework-specific env files). If valid values already exist, skip this step. - If the PostHog project token is missing, use the PostHog MCP server's
projects-gettool to retrieve the project'sapi_token. If multiple projects are returned, ask the user which project to use. If the MCP server is not connected or not authenticated, ask the user for their PostHog project token instead. - For the PostHog host URL: check the
projects-getMCP response for aregionfield —USmaps tohttps://us.i.posthog.com,EUmaps tohttps://eu.i.posthog.com. If the region is not available from the MCP response or from existing project configuration, ask the user: "Are you on PostHog US Cloud or EU Cloud?" Do not assume US Cloud. - Write these values to the appropriate env file using the framework's naming convention.
- Reference these environment variables in code instead of hardcoding them.
Reference files
references/openai.md- Openai observability installationreferences/azure-openai.md- Azure openai observability installationreferences/README.md- PostHog.ai for .netreferences/anthropic.md- Anthropic ai observability installationreferences/google.md- Google ai observability installationreferences/cohere.md- Cohere ai observability installationreferences/mistral.md- Mistral ai observability installationreferences/perplexity.md- Perplexity ai observability installationreferences/deepseek.md- Deepseek ai observability installationreferences/groq.md- Groq ai observability installationreferences/together-ai.md- Together ai observability installationreferences/fireworks-ai.md- Fireworks ai observability installationreferences/xai.md- Xai observability installationreferences/cerebras.md- Cerebras ai observability installationreferences/hugging-face.md- Hugging face ai observability installationreferences/ollama.md- Ollama ai observability installationreferences/openrouter.md- Openrouter ai observability installationreferences/langchain.md- Langchain ai observability installationreferences/llamaindex.md- Llamaindex ai observability installationreferences/crewai.md- Crewai observability installationreferences/autogen.md- Autogen ai observability installationreferences/dspy.md- Dspy ai observability installationreferences/langgraph.md- Langgraph ai observability installationreferences/pydantic-ai.md- Pydantic ai observability installationreferences/vercel-ai.md- Vercel ai SDK observability installationreferences/litellm.md- Litellm ai observability installationreferences/instructor.md- Instructor ai observability installationreferences/semantic-kernel.md- Semantic kernel ai observability installationreferences/mirascope.md- Mirascope ai observability installationreferences/mastra.md- Mastra ai observability installationreferences/smolagents.md- Smolagents ai observability installationreferences/openai-agents.md- Openai agents SDK observability installationreferences/portkey.md- Portkey ai observability installationreferences/helicone.md- Helicone ai observability installationreferences/manual-capture.md- Manual capture ai observability installationreferences/basics.md- Ai observability basicsreferences/traces.md- Tracesreferences/calculating-costs.md- Calculating llm costsreferences/COMMANDMENTS.md- Framework-specific rules the integration must follow
Each provider reference contains installation instructions, SDK setup, and code examples specific to that provider or framework. Find the reference that matches the user's stack.
If the user's provider isn't listed, use manual-capture.md as a fallback — it covers the generic event capture approach that works with any provider.
Key principles
- Environment variables: Always use environment variables for PostHog and LLM provider keys. Never hardcode them.
- Minimal changes: Add LLM analytics alongside existing LLM calls. Don't replace or restructure existing code.
- Trace all generations: Capture input tokens, output tokens, model name, latency, and costs for every LLM call.
- Link to users: Associate LLM generations with identified users via distinct IDs when possible.
- One provider at a time: Only instrument the provider(s) the user is actually using. Don't add instrumentation for providers not present in the codebase.
FAQ
What does the instrument-llm-analytics skill do?
Add PostHog AI Observability to trace AI model usage. Use after implementing LLM features or reviewing PRs to ensure all generations are captured with token counts, latency, and costs. Also handles initial PostHog SDK setup if not yet installed.
How do I install instrument-llm-analytics?
Run npx skills add https://github.com/posthog/ai-plugin --skill instrument-llm-analytics in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/instrument-llm-analytics/ (personal) or .claude/skills/instrument-llm-analytics/ (project).
Which AI agents can use instrument-llm-analytics?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is instrument-llm-analytics safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains instrument-llm-analytics?
It is published by posthog in posthog/ai-plugin, and it is an official skill from the maker of the technology it covers. It has 262 recorded installs.
More from posthog/ai-plugin
instrument-product-analytics
posthog/ai-plugin
Add PostHog product analytics events to track user behavior. Use after implementing new features or reviewing PRs to ensure meaningful user actions are captured. Also handles initial PostHog SDK setup if not yet installed.
instrument-feature-flags
posthog/ai-plugin
Add PostHog feature flags to gate new functionality. Use after implementing features or reviewing PRs to ensure safe rollouts with feature flag controls. Also handles initial PostHog SDK setup if not yet installed.
cleaning-up-stale-feature-flags
posthog/ai-plugin
Identify stale feature flags in a PostHog project and clean up the code that checks them. Use when the user wants to find, audit, or remove unused, fully rolled out, or abandoned feature flags, run a feature flag hygiene review or a weekly cleanup, reduce feature flag tech debt, or asks "which flags can I remove?". When the agent can read and edit a repository it performs the code cleanup itself: tested local changes, and one draft PR per flag when the user authorizes publishing. Agents without repository access generate a tailored cleanup prompt instead. Covers staleness detection, dependency checking, retained-path rules, and the code-first ordering. This skill does not archive or otherwise change a flag in PostHog.
auditing-experiments-flags
posthog/ai-plugin
Audit PostHog experiments and feature flags for configuration issues, staleness, and best-practice violations. Read when the user asks to audit, health-check, or review experiments or feature flags, check flag hygiene, or verify experiment setup.
querying-posthog-data
posthog/ai-plugin
Explains how to query PostHog data, defaulting to typed runners for supported product analytics. Read it before you write HogQL/SQL. Also read it before you call execute-sql against PostHog. Use it to find or aggregate PostHog entities. These entities include insights, dashboards, cohorts, feature flags, experiments, surveys, hog flows, warehouse data, and persons. Use it for trends, funnels, retention, lifecycle, paths, stickiness, web analytics, error tracking, logs, sessions, and LLM traces. Before you calculate a governed business or telemetry measure, check system.informationschema.metrics for an approved definition. Examples include MRR, activation, billable usage, active organizations, and failure rates. Use the approved definition before you derive a measure from raw events or use a typed domain tool. It also covers HogQL differences, system table schemas, functions, query examples, and schema discovery.
creating-experiments
posthog/ai-plugin
Guides agents through experiment creation: reading the project's setup with experiment-setup-context, defining the hypothesis, configuring rollout and bucketing, setting up analytics and running time, and reporting which choices are guesses. Delegates rollout decisions to configuring-experiment-rollout and metric setup to configuring-experiment-analytics. TRIGGER when: user asks to create a new experiment or A/B test, OR when you are about to call experiment-create. DO NOT TRIGGER when: user is updating an existing experiment, managing lifecycle, or only browsing experiments.
Popular ai & agents skills
google-agents-cli-publish
google/agents-cli
This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).
google-agents-cli-eval
google/agents-cli
This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
skill-creator
anthropics/skills
Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
codex-pet
prime-skills/runcomfy-agent-skills
Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.
full-output-enforcement
leonxlnx/taste-skill
Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.
executing-plans
obra/superpowers
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available