comparative-evaluation
A/B testing, side-by-side comparison, and preference ranking for AI outputs.
- Installs
- 128
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/owl-listener/ai-design-skills --skill comparative-evaluation- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads comparative-evaluation automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Comparative Evaluation
Absolute quality scores are useful but limited. Comparative evaluation — putting outputs side by side and asking which is better — often reveals quality differences that rubrics miss.
Comparison Methods
- A/B testing: Show different users different versions and compare outcomes
- Side-by-side evaluation: Show evaluators two outputs for the same input and ask which is better
- Preference ranking: Show evaluators multiple outputs and rank them from best to worst
- Paired comparison: Compare every pair of options to build a complete ranking
- Elo rating: Use tournament-style comparisons to develop continuous quality scores
Designing A/B Tests for AI
A/B testing AI is different from A/B testing UI:
- Variance is high: The same prompt can produce different outputs, so you need more samples
- Context matters: The same change might help for one task and hurt for another
- Metrics lag: AI quality changes may take time to show up in user behavior
- Interaction effects: A change to one part of the conversation affects all subsequent parts Design A/B tests with:
- Sufficient sample sizes to account for output variance
- Segmentation by task type and user experience level
- Multiple metrics (don't optimise for one at the expense of others)
- Guardrails to catch severe quality regressions quickly
Side-by-Side Evaluation Design
For human evaluation of AI outputs:
- Blind evaluation: Evaluators shouldn't know which version is which
- Consistent inputs: Compare outputs generated from the same input
- Structured criteria: Give evaluators specific dimensions to compare on, not just "which is better"
- Multiple evaluators: Use at least 3 evaluators per comparison for reliability
- Diverse inputs: Test across a representative sample of real user inputs
When to Use Comparative vs. Absolute Evaluation
- Comparative: Best for choosing between alternatives, detecting subtle quality differences, and model selection
- Absolute: Best for measuring against a standard, tracking progress over time, and certification
Design Artefacts
- A/B test design templates
- Side-by-side evaluation protocols
- Evaluator instructions and rubrics
- Sample size calculators for AI experiments
- Comparison result analysis frameworks
FAQ
What does the comparative-evaluation skill do?
A/B testing, side-by-side comparison, and preference ranking for AI outputs.
How do I install comparative-evaluation?
Run npx skills add https://github.com/owl-listener/ai-design-skills --skill comparative-evaluation in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/comparative-evaluation/ (personal) or .claude/skills/comparative-evaluation/ (project).
Which AI agents can use comparative-evaluation?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is comparative-evaluation safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains comparative-evaluation?
It is published by owl-listener in owl-listener/ai-design-skills. It has 128 recorded installs.
More from owl-listener/ai-design-skills
system-prompt-structure
owl-listener/ai-design-skills
Open the skill to read its SKILL.md instructions.
guardrail-design
owl-listener/ai-design-skills
Open the skill to read its SKILL.md instructions.
constraint-specification
owl-listener/ai-design-skills
Open the skill to read its SKILL.md instructions.
generative-ui
owl-listener/ai-design-skills
Open the skill to read its SKILL.md instructions.
conversation-patterns
owl-listener/ai-design-skills
Turn-taking, repair sequences, grounding, and dialogue structure for human-AI interaction.
progressive-disclosure
owl-listener/ai-design-skills
Open the skill to read its SKILL.md instructions.
Popular testing & qa skills
tdd
mattpocock/skills
Test-driven development. Use when the user wants to build features or fix bugs test-first, mentions "red-green-refactor", or wants integration tests.
grilling
mattpocock/skills
Grill the user relentlessly about a plan, decision, or idea. Use when the user wants to stress-test their thinking, or uses any 'grill' trigger phrases.
migrate-to-shoehorn
mattpocock/skills
Migrate test files from as type assertions to @total-typescript/shoehorn. Use when user mentions shoehorn, wants to replace as in tests, or needs partial test data.
run-train
lllllllama/rigorpilot-skills
Rigor Train skill for deep learning research repositories. Use when a documented or selected training command should be run conservatively for startup verification, short-run verification, full kickoff, or resume, with command, config, seed, log, checkpoint, status, and metric evidence written to standardized trainoutputs/. Do not use for environment setup, exploratory sweeps, speculative idea implementation, or end-to-end orchestration.
test-driven-development
obra/superpowers
Use when implementing any feature or bugfix, before writing implementation code
finishing-a-development-branch
obra/superpowers
Use when implementation is complete, all tests pass, and you need to decide how to integrate the work