MCPInstall

cli-eval

by cheaper-inferenceAI & Agents

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

Installs
988
Last 24h
+23
Trending rank
#5246
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/cheaper-inference/omniroute --skill cli-eval
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads cli-eval automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

<!-- generated by src/lib/agentSkills/generator.ts; manual edits will be overwritten -->

Overview

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

Quick install

npm install -g omniroute   # or: npx omniroute
omniroute --version

Subcommands

eval

Example:

omniroute eval

eval suites

Example:

omniroute eval suites

eval list

Example:

omniroute eval list

eval get <suiteId>

Example:

omniroute eval get <suiteId>

eval create

Flags:

  • --file <path>

Example:

omniroute eval create

eval run <suiteId>

Flags:

  • -m, --model <id>
  • --combo <name>
  • --concurrency <n>
  • --tag <tag>
  • --watch

Example:

omniroute eval run <suiteId>

eval list

Flags:

  • --suite <id>
  • --status <s>
  • --since <ts>
  • --limit <n>

Example:

omniroute eval list

eval get <runId>

Example:

omniroute eval get <runId>

eval results <runId>

Flags:

  • --failed

Example:

omniroute eval results <runId>

eval cancel <runId>

Flags:

  • --yes

Example:

omniroute eval cancel <runId>

eval scorecard <runId>

Example:

omniroute eval scorecard <runId>

simulate [prompt]

Flags:

  • --file <path>
  • -m, --model <id>
  • --combo <name>
  • --reasoning-effort <level>
  • --thinking-budget <n>
  • --explain

Example:

omniroute simulate [prompt]
<!-- skill:custom-start --> <!-- Migrated from skills/omniroute-cli-eval/SKILL.md (preserved curated content) -->

OmniRoute — CLI Evals

Requires the omniroute CLI. See CLI entry-point skill for install + global flags.

What are evals?

Evals are automated test suites that score LLM outputs against expected answers or rubrics. OmniRoute stores suites and run results in its local database.

Eval suites

omniroute eval suites list                       # List all eval suites
omniroute eval suites list --json                # JSON output

omniroute eval suites get <suiteId>              # Full suite definition

Create a suite

omniroute eval suites create \
  --name "code-quality" \
  --rubric "exact-match" \
  --samples-file ./samples.jsonl                 # JSONL: {input, expected_output}

Rubric options: exact-match, contains, llm-judge, regex.

--samples-file format (one JSON object per line):

{"input": "What is 2+2?", "expected_output": "4"}
{"input": "Translate 'hello' to Spanish", "expected_output": "hola"}

Run an eval

omniroute eval suites run <suiteId> \
  --model claude-sonnet-4-6                      # Run suite against a specific model

omniroute eval suites run <suiteId> \
  --model gpt-4o \
  --watch                                        # Live TUI progress (EvalWatch)

The run is asynchronous. Use --watch for a live terminal dashboard or poll manually:

RUN_ID=$(omniroute eval suites run <suiteId> --model claude-sonnet-4-6 --output json | jq -r '.id')
omniroute eval get $RUN_ID

Manage runs

omniroute eval list                              # List all eval runs
omniroute eval list --json

omniroute eval get <runId>                       # Run details (status, model, score)
omniroute eval results <runId>                   # Per-sample results
omniroute eval scorecard <runId>                 # Full scorecard with pass/fail per sample
omniroute eval cancel <runId>                    # Cancel a running eval

Scorecard output

omniroute eval scorecard <runId> --output json

Response fields per sample:

{
  "id": "sample-1",
  "score": 0.95,
  "passed": true,
  "input": "What is 2+2?",
  "output": "4",
  "expected": "4"
}

Comparing models

Run the same suite against multiple models and compare:

for MODEL in claude-sonnet-4-6 gpt-4o gemini-2.0-flash; do
  omniroute eval suites run $SUITE_ID --model $MODEL --output json | jq '{model: .model, score: .score}'
done

CI integration

# Run and fail CI if score drops below threshold
SCORE=$(omniroute eval suites run $SUITE_ID --model claude-sonnet-4-6 --output json | jq -r '.score')
python3 -c "import sys; score=float('$SCORE'); sys.exit(0 if score >= 0.90 else 1)"

Errors

  • suites create fails with invalid rubric → use one of: exact-match, contains, llm-judge, regex
  • suites run returns model not found → verify model ID with omniroute models --search <name>
  • eval get shows status: failed → check omniroute logs --search eval for error details
  • scorecard returns empty results → the run may still be running; poll omniroute eval get <runId> until status is completed
<!-- skill:custom-end -->

FAQ

What does the cli-eval skill do?

Create and run evaluation suites, watch live benchmark progress, view scorecards, compare model performance, and integrate eval runs with CI workflows from the CLI.

How do I install cli-eval?

Run npx skills add https://github.com/cheaper-inference/omniroute --skill cli-eval in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/cli-eval/ (personal) or .claude/skills/cli-eval/ (project).

Which AI agents can use cli-eval?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is cli-eval safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains cli-eval?

It is published by cheaper-inference in cheaper-inference/omniroute. It has 988 recorded installs.

More from cheaper-inference/omniroute

View all →

omni-combos-routing

cheaper-inference/omniroute

Create and manage routing combos with 19 strategies (priority, weighted, round-robin, Auto-combo, and more). Configure fallback chains, test routing outcomes, and retrieve combo metrics.

AI & Agents1.3K installs

cli-setup

cheaper-inference/omniroute

Run initial setup, configure global CLI settings, manage environment variables, check for updates, and configure autostart via the CLI setup and config commands.

AI & Agents1.2K installs

omni-mcp

cheaper-inference/omniroute

Connect to the OmniRoute MCP server (110 tools, 3 transports: SSE/stdio/HTTP). Covers routing, cache, compression, memory, skills, providers, and audit tools across 33 permission scopes.

AI & Agents1.1K installs

omni-providers

cheaper-inference/omniroute

Manage provider connections, API keys, OAuth flows, and connection tests via the REST API. List, add, update, remove, and test AI provider integrations across OmniRoute's 375-provider catalog.

Backend & APIs1.1K installs

cli-routing

cheaper-inference/omniroute

Create, list, update, and delete routing combos from the CLI. Test routing strategies, inspect combo metrics, and configure fallback chains interactively.

AI & Agents1.1K installs

omni-auth

cheaper-inference/omniroute

Manage API key authentication and session tokens. Start here to authenticate requests via Bearer token, obtain session cookies, and configure login requirements for the OmniRoute API.

Backend & APIs1.1K installs

Popular ai & agents skills

View category →

google-agents-cli-publish

google/agents-cli

This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).

AI & Agents513.9K installs

google-agents-cli-eval

google/agents-cli

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).

AI & Agents513.5K installs

skill-creator

anthropics/skills

Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.

OfficialAI & Agents416.4K installs

codex-pet

prime-skills/runcomfy-agent-skills

Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.

AI & Agents402.1K installs

full-output-enforcement

leonxlnx/taste-skill

Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.

AI & Agents401K installs

executing-plans

obra/superpowers

Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available

AI & Agents245.7K installs