MCPInstall

eval-creator-ci

by pskoettAI & Agents

[Beta] CI-only eval regression runner using gh-aw (GitHub Agentic Workflows). Runs all eval cases in .evals/ on a schedule or per-PR, reports pass/fail results, and can block merges on regressions. Also creates new eval cases from promoted patterns flagged by learning-aggregator-ci. Use when: you want automated regression testing of promoted rules in CI/headless pipelines. For interactive eval creation and runs, use eval-creator.

Installs
50
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/pskoett/pskoett-ai-skills --skill eval-creator-ci
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads eval-creator-ci automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Eval Creator CI

Install

gh skill install pskoett/pskoett-skills eval-creator-ci

For interactive sessions, use:

gh skill install pskoett/pskoett-skills eval-creator

Fallback using the Agent Skills CLI:

npx skills add pskoett/pskoett-skills/skills/eval-creator-ci
npx skills add pskoett/pskoett-skills/skills/eval-creator

Purpose

Runs the outer loop's regress-test step in CI. Executes all eval cases in .evals/, reports pass/fail results, and optionally blocks merges on regressions. Can also create new eval cases from promotion candidates flagged by learning-aggregator-ci.

The interactive eval-creator skill is designed for in-session use where the user creates evals and runs them with immediate feedback. This CI variant runs on schedule or per-PR and posts results as check annotations.

Context Limitation (Important)

CI agents do not have implementation context. They execute mechanical verification methods (grep checks, command checks, file checks, rule checks) defined in eval case files. They do not interpret results beyond pass/fail — nuanced judgment is left to human review of the posted report.

Prerequisites

  • GitHub Actions enabled on the repository
  • gh CLI authenticated with repo access
  • gh-aw extension installed (gh extension install github/gh-aw, v0.40.1+)
  • .evals/ directory with eval cases (created by eval-creator or eval-creator-ci)
  • .evals/EVAL_INDEX.md with eval case index

CI Contract

Hard rules for headless execution:

  1. Eval execution is read-only for code — eval cases read files and run check commands but do not modify source code
  2. Eval case creation writes to .evals/ only — when creating new evals from promotion candidates
  3. Headless — no interactive prompts, no approval gates
  4. Structured output — emit results as YAML under eval_creator_ci key
  5. Gate policy — can fail the check run on eval regressions (configurable)
  6. Single comment — post one consolidated results comment per run

Authoring Workflow (gh-aw)

  1. Copy references/workflow-example.md into .github/workflows/eval-creator-ci.md
  2. Customize trigger and gate policy
  3. Validate: gh aw compile (add --actionlint --zizmor for security scan)
  4. Push to enable

Persistence and Chaining

  • cache-memory: stores eval run history (last-run dates, result trends) across runs. Avoids re-running evals that haven't changed.
  • workflow_call: in create mode, triggered by learning-aggregator-ci via call-workflow. Receives promotion candidates as workflow inputs.
  • upload-artifact: persists eval results YAML for downstream consumption.

Workflow Rules

The CI agent follows these rules in order:

Mode: Run Evals (default)

  1. Read .evals/EVAL_INDEX.md to get the list of all eval cases
  2. For each eval case file in .evals/cases/: a. Read the eval case metadata and verification method b. Check preconditions — if not met, mark as skip c. Execute the verification method:
    • grep-check: Search target files for pattern, compare to expected (found/not_found)
    • command-check: Run the command, check exit code and/or output
    • file-check: Verify file or section exists
    • rule-check: Read target file, search for expected content d. Compare result to expected outcome e. Record pass/fail/skip
  3. Update .evals/EVAL_INDEX.md with last-run date and last-result for each case
  4. Emit structured YAML under key eval_creator_ci
  5. Post results summary as a PR comment or check annotation
  6. If gate policy is enabled and any eval fails: fail the check run

Mode: Create Evals (from promotion candidates)

  1. Read the learning_aggregator_ci artifact or gap report from the most recent learning-aggregator-ci run
  2. For each promotion-ready pattern with eval_candidate: true: a. Determine the appropriate verification method based on the pattern type b. Create the eval case file in .evals/cases/ with proper frontmatter c. Add the entry to .evals/EVAL_INDEX.md
  3. Commit the new eval cases (if running with write permissions)
  4. Report created evals in the output

Output Schema

eval_creator_ci:
  version: "0.1.0"
  source:
    run_id: "<workflow run ID>"
    trigger: "pull_request | schedule | workflow_dispatch"
    run_date: "YYYY-MM-DD"
  mode: "run | create | both"
  run_results:
    total: 12
    passed: 10
    failed: 1
    skipped: 1
    failures:
      - id: "eval-20260301-001"
        pattern_key: "harden.input_validation"
        rule_summary: "Always validate external inputs"
        expected: "not_found"
        actual: "found"
        target: "src/api/handler.ts"
        recovery_action: "Add input validation to new handler endpoint"
    skips:
      - id: "eval-20260315-003"
        reason: "Precondition not met: project does not use TypeScript"
  create_results:
    created: 2
    cases:
      - id: "eval-20260411-001"
        pattern_key: "simplify.dead_code"
        verification_method: "grep-check"
        source_learning: "LRN-20260301-001"
      - id: "eval-20260411-002"
        pattern_key: "harden.authorization"
        verification_method: "rule-check"
        source_learning: "LRN-20260315-003"
  summary:
    regressions: 1
    new_evals_created: 2
    gate_result: "fail"
    followup_required: true

Recommended Outputs

OutputDestinationContent
Eval resultsPR comment or check annotationPass/fail summary with failure details
YAML artifactWorkflow artifactMachine-readable eval_creator_ci payload
Check statusCheck runPass or fail based on gate policy
New eval files.evals/cases/ (if create mode)Eval case markdown files

Gate Policy

Configure blocking behavior:

PolicyBehavior
strictAny eval failure blocks the check run
advisoryFailures are reported but do not block
critical-onlyOnly evals from critical or high severity patterns block

Default: advisory (report but don't block). Teams should escalate to strict once eval coverage stabilizes.

Trigger Configuration

Recommended: per-PR + weekly schedule

on:
  pull_request:
    types: [opened, synchronize, reopened, ready_for_review]
  schedule:
    - cron: '0 10 * * 1'  # Monday 10am UTC (after learning-aggregator-ci)
  workflow_dispatch:

Per-PR runs catch regressions before merge. Weekly runs catch drift in the harness itself. Schedule after learning-aggregator-ci so new evals from promotions are available.

Integration with Other Skills

Upstream (feeds from)

  • eval-creator (interactive) — creates eval cases manually
  • learning-aggregator-ci — produces promotion candidates with eval_candidate: true
  • harness-updater (interactive) — flags eval candidates after promoting patterns

Downstream (feeds into)

  • self-improvement / self-improvement-ci — regression failures become new error entries in .learnings/
  • Human review — failure report posted for team triage
  • PR merge gate — can block merge on regressions (configurable)

Data Flow

learning-aggregator-ci → promotion candidates (eval_candidate: true)
                              ↓
                    eval-creator-ci (create mode)
                              ↓
                         .evals/cases/
                              ↓
                    eval-creator-ci (run mode, per-PR)
                              ↓
                    pass/fail report → PR comment + check annotation
                              ↓
                    regression failures → self-improvement-ci → .learnings/

Differences from Interactive Version

AspectInteractive (eval-creator)CI (eval-creator-ci)
TriggerManual invocationPR events, cron schedule, workflow_dispatch
Eval creationUser-driven with immediate feedbackAutomated from learning-aggregator-ci candidates
Eval executionIn-session with inline resultsHeadless with PR comment output
Human interactionUser reviews results inlineAsync review via GitHub
Gate behaviorNo blocking — informationalConfigurable: advisory, critical-only, strict
File modificationUpdates eval case metadataUpdates eval index + creates new cases (in create mode)

FAQ

What does the eval-creator-ci skill do?

[Beta] CI-only eval regression runner using gh-aw (GitHub Agentic Workflows). Runs all eval cases in .evals/ on a schedule or per-PR, reports pass/fail results, and can block merges on regressions. Also creates new eval cases from promoted patterns flagged by learning-aggregator-ci. Use when: you want automated regression testing of promoted rules in CI/headless pipelines. For interactive eval creation and runs, use eval-creator.

How do I install eval-creator-ci?

Run npx skills add https://github.com/pskoett/pskoett-ai-skills --skill eval-creator-ci in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/eval-creator-ci/ (personal) or .claude/skills/eval-creator-ci/ (project).

Which AI agents can use eval-creator-ci?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is eval-creator-ci safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains eval-creator-ci?

It is published by pskoett in pskoett/pskoett-ai-skills. It has 50 recorded installs.

More from pskoett/pskoett-ai-skills

View all →

self-improvement

pskoett/pskoett-ai-skills

Captures learnings, errors, corrections, and feature requests to enable continuous improvement. Use when: (1) User corrects Claude ('No, that's wrong...', 'Actually...'), (2) User requests a capability that doesn't exist, (3) Claude realizes its knowledge is outdated or incorrect, (4) A better approach is discovered for a recurring task, (5) Receiving a Handoff block from self-healing (a recurring verified heal with Recurrence-Count at least 3) to distill into a memory file or new skill. For ACTIVE runtime failures where the agent needs to apply and verify a fix mid-task, use self-healing instead (it files HEAL- entries with proof; self-improvement promotes accumulated patterns). Also review learnings before major tasks. For CI-only/headless learning capture, use self-improvement-ci.

DevOps & Cloud2K installs

self-improvement-ci

pskoett/pskoett-ai-skills

CI-only self-improvement workflow using gh-aw (GitHub Agentic Workflows). Captures recurring failure patterns and quality signals from pull request checks, emits structured learning candidates, and proposes durable prevention rules without interactive prompts. Use when: you want automated learning capture in CI/headless pipelines.

DevOps & Cloud545 installs

simplify-and-harden

pskoett/pskoett-ai-skills

Post-completion self-review for coding agents that runs simplify, harden, re-verification, and micro-documentation passes on non-trivial code or high-impact configuration changes. Use when a task is complete in a general agent session and you want a bounded quality and security sweep before signaling done. For CI pipeline execution, use simplify-and-harden-ci.

DevOps & Cloud535 installs

agent-teams-simplify-and-harden

pskoett/pskoett-ai-skills

Implementation + audit loop using parallel agent teams with structured simplify, harden, and document passes. Spawns implementation agents to do the work, then audit agents to find complexity, security gaps, and spec deviations, then loops until code compiles cleanly, all tests pass, and auditors find zero issues or the loop cap is reached. Use when: implementing features from a spec or plan, hardening existing code, fixing a batch of issues, or any multi-file task that benefits from a build-verify-fix cycle.

Code Quality & Workflow513 installs

plan-interview

pskoett/pskoett-ai-skills

Ensures alignment between user and Claude during feature/spec planning through a structured interview process. Use this skill when the user invokes /plan-interview before implementing a new feature, refactoring, or any non-trivial implementation task. The skill runs an upfront interview to gather requirements across technical constraints, scope boundaries, risk tolerance, and success criteria before any codebase exploration. Do NOT use this skill for: pure research/exploration tasks, simple bug fixes, or when the user just wants standard planning without the interview process.

Code Quality & Workflow485 installs

intent-framed-agent

pskoett/pskoett-ai-skills

Frames coding-agent work sessions with explicit intent capture and drift monitoring. Use when a session transitions from planning/Q&A to implementation for coding tasks, refactors, feature builds, bug fixes, or other multi-step execution where scope drift is a risk.

Code Quality & Workflow435 installs

Popular ai & agents skills

View category →

google-agents-cli-publish

google/agents-cli

This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).

AI & Agents513.9K installs

google-agents-cli-eval

google/agents-cli

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).

AI & Agents513.5K installs

skill-creator

anthropics/skills

Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.

OfficialAI & Agents416.4K installs

codex-pet

prime-skills/runcomfy-agent-skills

Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.

AI & Agents402.1K installs

full-output-enforcement

leonxlnx/taste-skill

Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.

AI & Agents401K installs

executing-plans

obra/superpowers

Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available

AI & Agents245.7K installs