trustworthy-experiments

Q: How do I install trustworthy-experiments?

Run npx skills add https://github.com/pmprompt/claude-plugin-product-management --skill trustworthy-experiments in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/trustworthy-experiments/ (personal) or .claude/skills/trustworthy-experiments/ (project).

by pmpromptTesting & QA

Use when asked to "run an A/B test", "design an experiment", "check statistical significance", "trust our results", "avoid false positives", or "experiment guardrails". Helps design, run, and interpret controlled experiments correctly. Based on Ronny Kohavi's framework from "Trustworthy Online Controlled Experiments".

Installs: 350
Last 24h: —
Trending rank: —
Security: Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/pmprompt/claude-plugin-product-management --skill trustworthy-experiments

01
Install
Run the command in your project. The CLI asks which agents to install for.
02
Describe the task
Ask your agent for something this skill covers. It loads trustworthy-experiments automatically when relevant.
03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Domain Context

This skill implements a proven product management framework. The approach combines best practices from industry leaders and is designed for practical application in day-to-day PM work.

Input Requirements

Context about your product, feature, or problem
Relevant data, research, or constraints (recommended but optional)
Clear articulation of what you're trying to achieve

Trustworthy Experiments

What It Is

Trustworthy Experiments is a framework for running controlled experiments (A/B tests) that produce reliable, actionable results. The core insight: most experiments fail, and many "successful" results are actually false positives.

The key shift: Move from "Did the experiment show a positive result?" to "Can I trust this result enough to act on it?"

Ronny Kohavi, who built experimentation platforms at Microsoft, Amazon, and Airbnb, found that:

66-92% of experiments fail to improve the target metric
8% of experiments have invalid results due to sample ratio mismatch alone
When the base success rate is 8%, a P-value of 0.05 still means 26% false positive risk

When to Use It

Use Trustworthy Experiments when you need to:

Design an A/B test that will produce valid, actionable results
Determine sample size and runtime for statistical power
Validate experiment results before making ship/no-ship decisions
Build an experimentation culture at your company
Choose metrics (OEC) that balance short-term gains with long-term value
Diagnose why results look suspicious (Twyman's Law)
Speed up experimentation without sacrificing validity

When Not to Use It

Don't use controlled experiments when:

You don't have enough users — Need tens of thousands minimum
The decision is one-time — Can't A/B test mergers or acquisitions
There's no real user choice — Employer-mandated software
You need immediate decisions — Experiments need time
The metric can't be measured — No experiment without observable outcomes

Resources

Book:

Trustworthy Online Controlled Experiments by Ronny Kohavi, Diane Tang, and Ya Xu

FAQ

What does the trustworthy-experiments skill do?

How do I install trustworthy-experiments?

Run npx skills add https://github.com/pmprompt/claude-plugin-product-management --skill trustworthy-experiments in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/trustworthy-experiments/ (personal) or .claude/skills/trustworthy-experiments/ (project).

Which AI agents can use trustworthy-experiments?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is trustworthy-experiments safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains trustworthy-experiments?

It is published by pmprompt in pmprompt/claude-plugin-product-management. It has 350 recorded installs.