sc-evaluate
LLM pipeline evaluation with oracle judge scoring. Runs prompts against gold standard datasets, evaluates output quality via LLM-as-judge, and generates scored reports with improvement recommendations.
- Installs
- 68
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/tony363/superclaude --skill sc-evaluate- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads sc-evaluate automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
LLM Evaluation Skill
Run LLM pipeline evaluation against gold standard datasets using oracle LLM-as-judge scoring. Measures output quality across weighted dimensions, identifies weak steps, and suggests prompt improvements.
Quick Start
# Full evaluation (all test cases, all steps)
/sc:evaluate
# Quick spot check
/sc:evaluate --cases=case_1,case_2 --steps=1,2,3
# Re-evaluate existing results without re-running pipeline
/sc:evaluate --skip-pipeline
# Generate outputs only (no evaluation)
/sc:evaluate --skip-eval
# Specify judge model
/sc:evaluate --judge-model=gpt-4o
# Dry run to preview plan
/sc:evaluate --dry-run
Behavioral Flow
- Discover - Find evaluation script, gold standards, and prompt files
- Configure - Parse scope (cases, steps, model overrides)
- Execute - Run pipeline on gold standard inputs
- Evaluate - Score outputs against gold standards via LLM-as-judge
- Analyze - Identify weak steps, dimension breakdowns, patterns
- Recommend - Suggest specific prompt improvements for low-scoring steps
- Report - Generate JSON + Markdown evaluation reports
Flags
| Flag | Type | Default | Description |
|---|---|---|---|
--cases | string | all | Comma-separated test case IDs to evaluate |
--steps | string | all | Comma-separated step numbers to evaluate |
--model | string | env default | Override pipeline model |
--judge-model | string | env default | Override judge/oracle model |
--skip-pipeline | bool | false | Skip pipeline execution, evaluate existing results |
--skip-eval | bool | false | Run pipeline only, skip evaluation |
--dry-run | bool | false | Preview execution plan without API calls |
--output | string | eval_runs/YYYYMMDD_HHMMSS/ | Output directory |
--concurrency | int | 5 | Parallel judge calls |
--threshold | int | 70 | Score threshold for "needs improvement" |
Phase 1: Discover Project Structure
Locate evaluation components:
| Component | Common Locations | Purpose |
|---|---|---|
| Evaluation script | scripts/run_eval.py, eval/run.py | Orchestrates pipeline + scoring |
| Gold standards | gold_standards/, test_data/, fixtures/ | Expected outputs |
| Prompts | prompts/, templates/ | Pipeline prompt templates |
| Rubrics | eval/rubrics.py, config/rubrics.yaml | Scoring dimensions and weights |
If no standard structure found, ask the user to specify paths.
Phase 2: Configure Scope
Parse arguments to determine:
- Which test cases to run (default: all discovered)
- Which pipeline steps to evaluate (default: all)
- Model overrides for pipeline and judge
- Output directory (default: timestamped)
Create output directory:
OUTPUT_DIR="${output:-eval_runs/$(date +%Y%m%d_%H%M%S)}"
mkdir -p "$OUTPUT_DIR"
Phase 3: Execute Pipeline
Run the pipeline on gold standard inputs:
python <eval_script> \
--output "$OUTPUT_DIR" \
--verbose \
[--cases CASES] \
[--steps STEPS] \
[--model MODEL] \
[--skip-pipeline] \
[--skip-eval]
API call estimation:
- Pipeline: steps x cases API calls
- Evaluation: scored_dimensions x cases judge calls
For quick validation, suggest running on 1-2 cases with 2-3 steps first.
Phase 4: Evaluate with LLM-as-Judge
For each step output, compare against gold standard using oracle LLM-as-judge:
Evaluation dimensions (customizable per project):
| Dimension | What It Measures |
|---|---|
| Content Agreement | Do outputs cover the same key points? |
| Structure Match | Is the organization/format similar? |
| Detail Accuracy | Are specific claims and data correct? |
| Completeness | Are all expected elements present? |
Each dimension has a weight (0.0-1.0) summing to 1.0 per step.
Phase 5: Analyze Results
Read and analyze evaluation report:
- Overall similarity score across all cases and steps
- Per-step scores — highlight any below threshold (default: 70/100)
- Per-case scores — identify consistently weak test cases
- Dimension breakdowns for weak steps
Score interpretation:
| Score Range | Assessment | Action |
|---|---|---|
| 85-100 | Excellent | No changes needed |
| 70-84 | Good | Minor tuning possible |
| 60-69 | Needs improvement | Prompt revision recommended |
| Below 60 | Poor | Prompt likely needs rewrite |
Phase 6: Recommend Improvements
For each step scoring below threshold:
- Read the current prompt template
- Read the gold standard output (expected)
- Read the pipeline output (actual)
- Compare and identify gaps:
- Missing instructions that gold standard captures
- Overly broad instructions causing divergent output
- Format/structure differences
- Specificity gaps
Present actionable suggestions:
### Step N: <step_name> (Score: XX/100)
**Weakest Dimension**: <dimension> (XX/100)
**Gap Analysis**:
- Gold standard includes <X> but prompt doesn't instruct it
- Output format diverges: gold uses <format>, output uses <other>
**Suggested Prompt Changes**:
1. Add instruction: "<specific instruction>"
2. Clarify format: "<format guidance>"
3. Add example: "<example output snippet>"
Output Structure
eval_runs/YYYYMMDD_HHMMSS/
results/ # Pipeline outputs
case_1/
step_01_<name>.md
step_02_<name>.md
...
case_2/
...
evaluation/ # Judge scores
evaluation_report.json
evaluation_report.md
per_step_scores.csv
per_case_scores.csv
MCP Integration
PAL MCP (Optional)
| Tool | When | Purpose |
|---|---|---|
mcp__pal__thinkdeep | Low-scoring steps | Deep analysis of why outputs diverge |
mcp__pal__consensus | Prompt revision | Multi-model validation of proposed changes |
mcp__pal__codereview | Eval script | Review evaluation pipeline code |
Rube MCP (Optional)
| Tool | When | Purpose |
|---|---|---|
mcp__rube__RUBE_REMOTE_WORKBENCH | Large eval runs | Process results in Python sandbox |
mcp__rube__RUBE_MULTI_EXECUTE_TOOL | Notifications | Report results to Slack/email |
Error Handling
| Scenario | Action |
|---|---|
| No eval script found | Ask user for script path |
| No gold standards found | Ask user for gold standard directory |
| API rate limit | Reduce concurrency, add delays |
| Pipeline step fails | Log error, continue with remaining steps |
| Judge returns invalid score | Retry once, then flag for manual review |
| Output directory exists | Append timestamp suffix |
Guardrails
- Always pass
--verbosefor progress visibility - Warn about API call counts before full runs
- Suggest quick validation on subset before full evaluation
- Preserve all intermediate outputs for debugging
- Never modify gold standard files
Tool Coordination
- Bash - Run evaluation scripts
- Read - Inspect prompts, gold standards, outputs, reports
- Write - Generate reports
- Grep - Search for patterns in outputs
- PAL MCP - Deep analysis of score gaps
FAQ
What does the sc-evaluate skill do?
LLM pipeline evaluation with oracle judge scoring. Runs prompts against gold standard datasets, evaluates output quality via LLM-as-judge, and generates scored reports with improvement recommendations.
How do I install sc-evaluate?
Run npx skills add https://github.com/tony363/superclaude --skill sc-evaluate in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/sc-evaluate/ (personal) or .claude/skills/sc-evaluate/ (project).
Which AI agents can use sc-evaluate?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is sc-evaluate safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains sc-evaluate?
It is published by tony363 in tony363/superclaude. It has 68 recorded installs.
More from tony363/superclaude
agent-performance-engineer
tony363/superclaude
Expert performance engineer specializing in system optimization, bottleneck identification, and scalability engineering. Masters performance testing, profiling, and tuning across applications, databases, and infrastructure with focus on achieving optimal response times and resource efficiency.
agent-security-engineer
tony363/superclaude
Expert infrastructure security engineer specializing in DevSecOps, cloud security, and compliance frameworks. Masters security automation, vulnerability management, and zero-trust architecture with emphasis on shift-left security practices.
sc-estimate
tony363/superclaude
Provide development estimates for tasks, features, or projects with intelligent analysis. Use when planning timelines, assessing complexity, or scoping resources.
agent-fullstack-developer
tony363/superclaude
End-to-end feature owner with expertise across the entire stack. Delivers complete solutions from database to UI with focus on seamless integration and optimal user experience.
sc-design
tony363/superclaude
Design system architecture, APIs, and component interfaces with comprehensive specifications. Use when planning architecture, designing APIs, creating component interfaces, or modeling databases.
agent-technical-writer
tony363/superclaude
Open the skill to read its SKILL.md instructions.
Popular ai & agents skills
google-agents-cli-publish
google/agents-cli
This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).
google-agents-cli-eval
google/agents-cli
This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
skill-creator
anthropics/skills
Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
codex-pet
prime-skills/runcomfy-agent-skills
Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.
full-output-enforcement
leonxlnx/taste-skill
Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.
executing-plans
obra/superpowers
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available