add-runner-eval
Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
- Installs
- 40
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/paperclipai/paperclip --skill add-runner-eval- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads add-runner-eval automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
Add a Runner Eval
Use this skill for the Runner Evals family: a real Runner/provider session against a seeded mock control plane. Product browser/server/database/Daytona coverage belongs in add-product-e2e-eval.
Locate the Paperclip checkout using PAPERCLIP_ROOT when supplied, or
git rev-parse --show-toplevel from a checkout. From outside Git, inspect the
workspace roots (for example ~/paperclipai/paperclip) and verify that the
selected root contains packages/paperclip-runner and tests/runner-e2e.
Locate paperclip-evals using PAPERCLIP_EVALS_ROOT or a discovered sibling;
a worktree's parent directory need not contain that repository. Read
doc/evals.md and packages/paperclip-runner/docs/runner-protocol-live-evals.md,
then inspect the nearest existing case, roster, schema, and report test before
editing. Definitions and authored cases belong in the sibling
paperclip-evals/evals/paperclip-runner; Runner integration, aggregation,
viewer, and publication behavior belongs in packages/paperclip-runner.
Keep the control-plane boundary explicit in names and documentation.
The sibling eval README is the concrete map: cases live under cases/,
company fixtures under fixtures/, runtime/model settings under configs/,
selections under rosters/, and maintained campaign membership under
campaigns/live-direct-full.json. Update inventory/coverage mappings when the
program requires them; a new file alone does not join the maintained campaign.
From the Evals repository root, adapt these provider-free checks to the case
and roster you changed. Run the reliability-plan validator only when that
separate plan changes:
python3 evals/paperclip-runner/tools/eval_program.py validate \
--case evals/paperclip-runner/cases/get-task-context.json \
--config evals/paperclip-runner/configs/live-codex-pinned.json
python3 evals/paperclip-runner/tools/run_live_roster.py validate \
--roster evals/paperclip-runner/rosters/live-mini.json --run-id validate-new-case
python3 evals/paperclip-runner/tools/run_live_campaign.py validate \
--campaign evals/paperclip-runner/campaigns/live-direct-full.json
python3 evals/paperclip-runner/tools/reliability_campaign.py validate \
--plan evals/paperclip-runner/campaigns/paperclip-runner-reliability.json
Use nearby positive and negative grader cases/fixtures to calibrate the new assertion, including malformed or missing evidence where the grader must fail closed. Preserve the existing machine disposition and grade; product, model/provider, grading, and infrastructure labels are analytical annotations, not instructions to rewrite classifiers.
Author one bounded case with a deterministic semantic assertion and an inspectable result. Declare its expected operation, state effect, provider lane/profile, timeout and retry policy, and any required evidence. Do not grade hidden reasoning, infer success from a provider terminal message, or invent conversation/tool evidence. Public output follows the reviewed projection: sanitized fixture conversation and allowlisted tool outcomes may be published; raw trusted artifacts, credentials, secrets, private references, and hidden reasoning may not.
Validate without provider calls first using the commands above and the relevant
report/render validation documented in the Runner docs. When a live run is
authorized, pin the Paperclip commit and exact 40-character
paperclip-evals commit, select the smallest useful roster, and retain the
complete provenance and cost record.
Update authoritative detailed docs when the contract or command changes, then
link from doc/evals.md rather than duplicating the Runner runbook. Keep public
reports immutable and use the reviewed projection; sanitized fixture
conversation and allowlisted tool outcomes may be public, while credentials,
secrets, private references, raw trusted payloads, and hidden reasoning must not
be exposed.
FAQ
What does the add-runner-eval skill do?
Add or extend a Paperclip Runner protocol evaluation definition, roster, assertion, or report fixture with provenance and narrow validation.
How do I install add-runner-eval?
Run npx skills add https://github.com/paperclipai/paperclip --skill add-runner-eval in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/add-runner-eval/ (personal) or .claude/skills/add-runner-eval/ (project).
Which AI agents can use add-runner-eval?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is add-runner-eval safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains add-runner-eval?
It is published by paperclipai in paperclipai/paperclip. It has 40 recorded installs.
More from paperclipai/paperclip
paperclip
paperclipai/paperclip
Use for Paperclip-managed tasks and heartbeats: reading task context, delivering task documents or files, updating completion or blockers, coordinating or delegating work, and following company governance. Includes control plane API operations for assignments, comments, approvals, and routines.
paperclip-create-agent
paperclipai/paperclip
Create new agents in Paperclip with governance-aware hiring. Use when you need to inspect adapter configuration options, compare existing agent configs, draft a new agent prompt/config, and submit a hire request.
paperclip-create-plugin
paperclipai/paperclip
Create and develop external Paperclip plugins with the CLI-first workflow. Use when scaffolding a plugin, iterating on a local plugin, installing it into Paperclip, or updating plugin authoring docs.
para-memory-files
paperclipai/paperclip
Use a file-based PARA memory system to store, retrieve, and organize durable knowledge across sessions. Trigger on saving facts, daily notes, entity records, weekly synthesis, recall, tacit user patterns, or plan memory.
create-agent-adapter
paperclipai/paperclip
Open the skill to read its SKILL.md instructions.
Popular ai & agents skills
google-agents-cli-publish
google/agents-cli
This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).
google-agents-cli-eval
google/agents-cli
This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
skill-creator
anthropics/skills
Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
codex-pet
prime-skills/runcomfy-agent-skills
Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.
full-output-enforcement
leonxlnx/taste-skill
Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.
executing-plans
obra/superpowers
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available