a-b-test-design
Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use test-scenario.
- Installs
- 2.1K
- Last 24h
- +24
- Trending rank
- #5061
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/owl-listener/designer-skills --skill a-b-test-design- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads a-b-test-design automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
A/B Test Design
You are an expert in designing rigorous A/B experiments that produce actionable results.
What You Do
You design A/B tests with clear hypotheses, controlled variants, appropriate metrics, and statistical rigor.
Test Structure
1. Hypothesis
Structured as: 'If we [change], then [outcome] will [improve/decrease] because [rationale].'
2. Variants
- Control (A): current design
- Treatment (B): proposed change
- Keep changes isolated — test one variable at a time
3. Primary Metric
The single most important measure of success. Must be measurable, relevant, and sensitive to the change.
4. Secondary Metrics
Supporting measures and guardrail metrics to detect unintended consequences.
5. Sample Size
Based on: minimum detectable effect, baseline conversion rate, statistical significance level (typically 95%), and power (typically 80%).
6. Duration
Run until sample size is reached. Account for weekly cycles (run in full weeks). Minimum 1-2 weeks typically.
Common Pitfalls
- Peeking at results before completion
- Too many variants at once
- Metric not sensitive enough to detect change
- Sample size too small
- Not accounting for novelty effects
- Ignoring segmentation effects
When Not to A/B Test
- Very low traffic (insufficient sample)
- Ethical concerns with withholding improvement
- Foundational changes that affect everything
- When qualitative insight is more valuable
Best Practices
- One hypothesis per test
- Document everything before starting
- Don't stop early on positive results
- Analyze segments after overall results
- Share learnings broadly regardless of outcome
FAQ
What does the a-b-test-design skill do?
Design an A/B experiment — hypothesis, variants, primary metric, and sample size. Use when a change can be measured quantitatively at scale. For observing behaviour qualitatively, use test-scenario.
How do I install a-b-test-design?
Run npx skills add https://github.com/owl-listener/designer-skills --skill a-b-test-design in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/a-b-test-design/ (personal) or .claude/skills/a-b-test-design/ (project).
Which AI agents can use a-b-test-design?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is a-b-test-design safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains a-b-test-design?
It is published by owl-listener in owl-listener/designer-skills. It has 2,081 recorded installs.
More from owl-listener/designer-skills
presentation-deck
owl-listener/designer-skills
Structure a design presentation for a specific audience and decision. Use when presenting internally. For a portfolio narrative use case-study; for the written argument use design-rationale.
typography-scale
owl-listener/designer-skills
Create a modular type scale with size, weight, and line-height relationships. Use when establishing typographic structure. For line length only use readable-measure; for judging type on an existing screen use critique-typography (visual-critique).
illustration-style
owl-listener/designer-skills
Define an illustration style guide — visual language, colour usage, and application rules. Use when commissioning or standardising illustration. For icons, use icon-system (design-systems).
visual-hierarchy
owl-listener/designer-skills
Establish hierarchy through size, weight, colour, spacing, and position so the eye lands in the intended order. Use when composing new work. For judging an existing screen, use critique-visual-hierarchy (visual-critique).
color-system
owl-listener/designer-skills
Build a product colour system — tonal scales, semantic roles, and contrast compliance. Use when defining or rebuilding colour from scratch. For dark-mode adaptation use dark-mode-design; for chart palettes use data-visualization; for multi-brand token architecture use theming-system (design-systems).
dark-mode-design
owl-listener/designer-skills
Adapt an existing palette to dark mode — surface elevation, contrast rebalancing, and desaturation rules. Use when you already have a light palette to translate. For building the base palette first, use color-system.
Popular testing & qa skills
tdd
mattpocock/skills
Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.
grilling
mattpocock/skills
Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thinking, or uses any 'grill' trigger phrases.
migrate-to-shoehorn
mattpocock/skills
Migrate test files from as type assertions to @total-typescript/shoehorn. Use when user mentions shoehorn, wants to replace as in tests, or needs partial test data.
run-train
lllllllama/rigorpilot-skills
Rigor Train skill for deep learning research repositories. Use when a documented or selected training command should be run conservatively for startup verification, short-run verification, full kickoff, or resume, with command, config, seed, log, checkpoint, status, and metric evidence written to standardized trainoutputs/. Do not use for environment setup, exploratory sweeps, speculative idea implementation, or end-to-end orchestration.
test-driven-development
obra/superpowers
Use when implementing any feature or bugfix, before writing implementation code
finishing-a-development-branch
obra/superpowers
Use when implementation is complete, all tests pass, and you need to decide how to integrate the work