MCPInstall

eval-writer

by langchain-ai OfficialAI & Agents

Create new eval suites for the deepagentsjs monorepo. Handles dataset design, test case scaffolding, scoring logic, vitest configuration, and LangSmith integration. Use when the user asks to: (1) create an eval, (2) write an evaluation, (3) add a benchmark, (4) build an eval suite, (5) evaluate agent behaviour, (6) add test cases for a capability, or (7) implement an existing benchmark (e.g. oolong, AgentBench, SWE-bench). Trigger on phrases like 'create eval', 'new eval', 'add eval', 'benchmark', 'evaluate', 'eval suite', 'write evals for'.

Installs
51
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/langchain-ai/deepagentsjs --skill eval-writer
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads eval-writer automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

Eval Writer

Create new eval suites for the deepagentsjs monorepo. Each eval is an independent workspace package under evals/ that uses the @deepagents/evals harness, runs via vitest, and reports results to LangSmith.

Before you start

Read the existing eval infrastructure to understand current patterns:

internal/eval-harness/src/index.ts    # EvalRunner, RunAgentParams, matchers
internal/eval-harness/src/deepagent.ts # DeepAgentEvalRunner, extend()
internal/eval-harness/src/setup.ts     # Registered runners
evals/README.md                        # User-facing docs
internal/eval-harness/README.md        # Harness internals

Scan existing evals for conventions:

evals/basic/index.test.ts     # Simple: system prompt, reasoning
evals/files/index.test.ts     # File ops: read, write, edit, glob, grep
evals/subagents/index.test.ts # Delegation: task tool, named subagents

Workflow

1. Understand the eval requirements

Clarify with the user:

  • What capability is being evaluated? (file ops, tool use, multi-turn reasoning, memory, code generation, etc.)
  • Where do test cases come from? Options:
    • Inline — hardcoded in the test file (simple, good for <20 cases)
    • JSON/JSONL fixture — checked into the eval package (good for 20-200 cases)
    • External dataset — downloaded at setup time (good for published benchmarks)
    • LangSmith dataset — pulled from LangSmith API (good for collaborative curation)
  • How should results be scored? Options:
    • Trajectory matchers — step count, tool calls, final text (built-in)
    • Exact/fuzzy match — compare output to reference (simple)
    • LLM-as-judge — use a model to grade the output (complex evals)
    • Code execution — run generated code and check results (SWE-bench style)
    • Custom evaluator — domain-specific scoring function
  • Does the agent need special configuration? (custom tools, subagents, system prompt, initial files)

2. Create the eval package

Every eval is a workspace package under evals/<name>/.

Directory structure

evals/<name>/
├── package.json
├── vitest.config.ts
├── index.test.ts
├── README.md
└── (optional) fixtures/       # JSON/JSONL test data
└── (optional) vitest.setup.ts  # Dataset loading, custom setup
└── (optional) evaluators.ts   # Custom scoring functions

package.json

{
  "name": "@deepagents/eval-<name>",
  "private": true,
  "type": "module",
  "scripts": {
    "test:eval": "vitest run"
  },
  "dependencies": {
    "@deepagents/evals": "workspace:*",
    "deepagents": "workspace:*",
    "langsmith": "^0.5.4",
    "vitest": "^4.0.18"
  }
}

Add extra dependencies as needed (e.g. zod for tool schemas, langchain for tool() helper, dataset-specific packages).

vitest.config.ts

import { defineConfig } from "vitest/config";

export default defineConfig({
  test: {
    environment: "node",
    globals: false,
    testTimeout: 120_000,
    hookTimeout: 60_000,
    teardownTimeout: 60_000,
    include: ["**/*.test.ts"],
    setupFiles: ["@deepagents/evals/setup"],
    reporters: ["default", "langsmith/vitest/reporter"],
  },
});

Adjust testTimeout for long-running evals (multi-turn, code execution). Add "./vitest.setup.ts" to setupFiles if the eval needs custom setup (dataset loading, etc.).

README.md

# <name>

<One-line description of what this eval tests.>

Verify workspace registration

Check that pnpm-workspace.yaml includes "evals/*". It should already be there — if not, add it.

3. Design test cases

Pattern A: Inline test cases (simple evals)

Best for small, hand-crafted test suites. Each test is an ls.test() call with inputs and optional referenceOutputs.

ls.test(
  "descriptive test name",
  {
    inputs: { query: "What is 2+2?" },
    referenceOutputs: { expectedAnswer: "4" },
  },
  async ({ inputs, referenceOutputs }) => {
    const result = await runner.run({ query: inputs.query });
    // assertions...
  },
);

Pattern B: Data-driven with ls.test.each (medium evals)

Best for 10-200 cases from a fixture file. Load the data and iterate:

import testCases from "./fixtures/cases.json";

// testCases = [{ inputs: { query: "..." }, referenceOutputs: { answer: "..." } }, ...]

ls.test.each(testCases)(
  "case: ${inputs.query}",
  async ({ inputs, referenceOutputs }) => {
    const result = await runner.run({ query: inputs.query });
    // assertions using referenceOutputs...
  },
);

The fixture JSON must be an array of objects with at minimum { inputs: {...} }. Optional fields: referenceOutputs, id, metadata, split.

Pattern C: External dataset (published benchmarks)

For published benchmarks (oolong, AgentBench, SWE-bench, etc.), download and cache the dataset in a setup file.

Create vitest.setup.ts:

import { existsSync, mkdirSync, writeFileSync, readFileSync } from "fs";
import { join } from "path";

const CACHE_DIR = join(import.meta.dirname, ".cache");
const DATA_PATH = join(CACHE_DIR, "dataset.json");

export async function loadDataset(): Promise<TestCase[]> {
  if (existsSync(DATA_PATH)) {
    return JSON.parse(readFileSync(DATA_PATH, "utf-8"));
  }

  mkdirSync(CACHE_DIR, { recursive: true });

  // Download from source — adapt to the specific benchmark
  const response = await fetch("https://example.com/dataset.json");
  const data = await response.json();

  // Transform into eval format
  const cases = data.map((item: any) => ({
    inputs: { query: item.question },
    referenceOutputs: { answer: item.gold_answer },
    metadata: { source: item.id, category: item.category },
  }));

  writeFileSync(DATA_PATH, JSON.stringify(cases, null, 2));
  return cases;
}

Add .cache/ to .gitignore in the eval package.

Then register it as a vitest setup file in vitest.config.ts:

setupFiles: ["@deepagents/evals/setup", "./vitest.setup.ts"],

And in the test file:

import { loadDataset } from "./vitest.setup.js";

const dataset = await loadDataset();

ls.describe(runner.name, () => {
  ls.test.each(dataset)(
    "${metadata.source}: ${inputs.query}",
    async ({ inputs, referenceOutputs }) => {
      // ...
    },
  );
}, { projectName: "deepagents-js-<name>", upsert: true });

Pattern D: LangSmith dataset

Pull test cases from a LangSmith dataset. Useful for collaborative curation where non-engineers add examples via the LangSmith UI.

import { Client } from "langsmith";

const client = new Client();

export async function loadDataset(): Promise<TestCase[]> {
  const examples = [];
  for await (const example of client.listExamples({
    datasetName: "my-dataset-name",
  })) {
    examples.push({
      id: example.id,
      inputs: example.inputs,
      referenceOutputs: example.outputs ?? {},
    });
  }
  return examples;
}

4. Write scoring logic

Built-in trajectory matchers

The harness provides vitest matchers that also log LangSmith feedback scores. Use these as the primary building blocks:

// Exact step count
expect(result).toHaveAgentSteps(3);

// Exact tool-call count across all steps
expect(result).toHaveToolCallRequests(2);

// Check a specific tool call in step N (1-indexed)
expect(result).toHaveToolCallInStep(1, {
  name: "write_file",
  argsContains: { file_path: "/out.txt" },  // partial match
  argsEquals: { file_path: "/out.txt" },     // exact match
});

// Final response text
expect(result).toHaveFinalTextContaining("hello", true /* caseInsensitive */);

// Extract final text for custom assertions
import { getFinalText } from "@deepagents/evals";
const text = getFinalText(result);
expect(text.trim()).toBe("4");

// File system assertions
expect(result.files["/output.md"]).toContain("expected content");
expect(Object.keys(result.files)).toHaveLength(3);

Custom feedback logging

Log additional LangSmith feedback scores beyond what matchers provide:

import * as ls from "langsmith/vitest";

// Numeric score
ls.logFeedback({ key: "accuracy", score: 0.95 });

// Boolean score
ls.logFeedback({ key: "correct", score: 1 });

// With comment
ls.logFeedback({ key: "quality", score: 0.8, comment: "Minor formatting issue" });

LLM-as-judge evaluators

For subjective quality, use ls.wrapEvaluator() to create a traced evaluator that logs feedback automatically:

import * as ls from "langsmith/vitest";
import { ChatAnthropic } from "@langchain/anthropic";

const judge = new ChatAnthropic({ model: "claude-sonnet-4-5-20250929" });

const evaluateHelpfulness = ls.wrapEvaluator(
  async ({ inputs, outputs, referenceOutputs }) => {
    const response = await judge.invoke([
      {
        role: "system",
        content: `Rate the helpfulness of the assistant's response on a scale of 0-1.
          Respond with JSON: { "score": <number>, "reasoning": "<explanation>" }`,
      },
      {
        role: "user",
        content: `Question: ${inputs.query}\nExpected: ${referenceOutputs.answer}\nActual: ${outputs.response}`,
      },
    ]);

    const parsed = JSON.parse(response.content as string);
    return {
      key: "helpfulness",
      score: parsed.score,
      comment: parsed.reasoning,
    };
  },
);

// In a test:
const result = await runner.run({ query: inputs.query });
const text = getFinalText(result);
await evaluateHelpfulness({
  inputs: { query: inputs.query },
  outputs: { response: text },
  referenceOutputs: referenceOutputs ?? {},
});

5. Wire up the test file

Minimal template

import * as ls from "langsmith/vitest";
import { expect } from "vitest";
import { getDefaultRunner, getFinalText } from "@deepagents/evals";

const runner = getDefaultRunner();

ls.describe(
  runner.name,
  () => {
    ls.test(
      "test name",
      {
        inputs: { query: "..." },
        referenceOutputs: { answer: "..." },
      },
      async ({ inputs, referenceOutputs }) => {
        const result = await runner.run({ query: inputs.query });

        expect(result).toHaveAgentSteps(1);
        expect(result).toHaveFinalTextContaining(referenceOutputs.answer);
      },
    );
  },
  { projectName: "deepagents-js-<name>", upsert: true },
);

Key conventions

  • getDefaultRunner() — reads EVAL_RUNNER env var. Throws if not set.
  • runner.name — used as ls.describe name → becomes the LangSmith dataset name.
  • runner.run({ query, initialFiles? }) — pure invocation. Returns AgentTrajectory.
  • runner.extend({ systemPrompt?, tools?, subagents?, ... }) — returns a new runner with agent config overrides. Use for tests that need custom agent setup.
  • projectName in ls.describe config — sets the LangSmith project for tracing. Convention: "deepagents-js-<eval-name>".
  • upsert: true — reuse existing dataset/project instead of creating new ones each run.
  • Always import expect from vitest — the harness extends it with custom matchers at import time.
  • ls.logOutputs() is called inside the runner — do NOT call it in test code.

Using extend() for custom agent config

// Custom system prompt
const result = await runner
  .extend({ systemPrompt: "You are a code reviewer." })
  .run({ query: inputs.query });

// Custom tools
const result = await runner
  .extend({ tools: [myCustomTool] })
  .run({ query: inputs.query });

// Custom subagents
const result = await runner
  .extend({
    subagents: [{
      name: "researcher",
      description: "Research assistant",
      systemPrompt: "You help with research.",
      tools: [searchTool],
    }],
  })
  .run({ query: inputs.query });

Using initialFiles for seeded state

const result = await runner.run({
  query: "Read /data.csv and count the rows.",
  initialFiles: {
    "/data.csv": "name,age\nAlice,30\nBob,25\n",
  },
});

Sandbox-backed evals (containerized execution)

The default eval runners use the in-memory StateBackend — the agent can read/write files but cannot execute shell commands, install packages, or interact with a real OS. This is fine for testing tool selection, reasoning, and file operations.

For evals that need real execution (SWE-bench, code generation, agentic benchmarks), the agent must run against a sandbox backend. Available sandbox providers:

ProviderPackageUse case
Modal@deepagents/modalRemote containers, GPU support
Daytona@deepagents/daytonaCloud dev environments
Deno@deepagents/denoLightweight local sandboxes
Node VFS@deepagents/node-vfsIn-process virtual filesystem + shell

Pass the sandbox via extend({ backend }). Manage its lifecycle with beforeAll / afterAll (suite-level) or beforeEach / afterEach (per-test isolation):

import * as ls from "langsmith/vitest";
import { expect, beforeAll, afterAll } from "vitest";
import { getDefaultRunner, getFinalText } from "@deepagents/evals";
import { ModalSandbox } from "@deepagents/modal";

const runner = getDefaultRunner();
let sandbox: ModalSandbox;

beforeAll(async () => {
  sandbox = await ModalSandbox.create({
    image: "python:3.12-slim",
    timeout: 600,
  });
});

afterAll(async () => {
  await sandbox?.terminate();
});

ls.describe(
  runner.name,
  () => {
    ls.test(
      "agent can run python",
      { inputs: { query: "Write a Python script that prints 'hello' and run it." } },
      async ({ inputs }) => {
        const result = await runner
          .extend({ backend: sandbox })
          .run({ query: inputs.query });

        expect(result).toHaveFinalTextContaining("hello");
      },
    );
  },
  { projectName: "deepagents-js-sandbox-eval", upsert: true },
);

For per-test isolation (each test gets a fresh sandbox):

import { beforeEach, afterEach } from "vitest";

let sandbox: ModalSandbox;

beforeEach(async () => {
  sandbox = await ModalSandbox.create({ image: "python:3.12-slim" });
});

afterEach(async () => {
  await sandbox?.terminate();
});

When to containerize:

  • The eval requires execute() (shell commands)
  • The eval involves installing packages or modifying system state
  • The eval runs untrusted or generated code
  • Tests need filesystem isolation from each other

When in-memory is fine:

  • Testing tool selection, reasoning, or response quality
  • File read/write/edit operations (handled by StateBackend + initialFiles)
  • System prompt adherence, subagent routing

Add the sandbox provider to package.json dependencies:

{
  "dependencies": {
    "@deepagents/modal": "workspace:*"
  }
}

And increase testTimeout in vitest.config.ts — sandbox creation adds overhead:

testTimeout: 300_000,  // 5 minutes for sandbox evals
hookTimeout: 120_000,  // sandbox setup/teardown

6. Install and verify

# From repo root
pnpm install

# Build the harness (if you changed it)
cd internal/eval-harness && pnpm build && cd ../..

# Run the new eval
EVAL_RUNNER=sonnet-4-5 pnpm --filter @deepagents/eval-<name> test:eval

7. Update documentation

Add the new eval to evals/README.md in the "Available eval suites" table:

| [`<name>/`](./<name>/) | <one-line description> |

Parity with Python deepagents evals

The Python deepagents package has eval suites in libs/deepagents/tests/evals/. The JS evals should maintain parity. When creating a new eval, check the Python source at https://github.com/langchain-ai/deepagents/blob/v0.5/libs/deepagents/tests/evals/ for the reference implementation.

Current parity status

Python evalJS evalStatus
test_system_prompt.pyevals/basic/✅ Covered
test_file_operations.pyevals/files/✅ Covered
test_subagents.pyevals/subagents/✅ Covered
test_memory.pyevals/memory/✅ Covered
test_hitl.pyevals/hitl/✅ Covered
test_skills.pyevals/skills/✅ Covered
test_summarization.pyevals/summarization/❌ Missing

Notes on HITL evals

HITL evals require multi-step invocation (invoke → check interrupts → resume with Command). The eval runner's run() does a single invocation, so HITL tests construct agents directly via createDeepAgent() with a checkpointer and interruptOn config. See evals/hitl/index.test.ts for the pattern.

Notes on summarization evals

Summarization evals need SummarizationMiddleware with low token thresholds, a checkpointer, a real/virtual filesystem backend, and multi-turn invocations. These tests would bypass the standard EvalRunner and construct agents directly, similar to HITL.

Reference: ls.test.each API

ls.test.each is the most powerful pattern for data-driven evals. The table must be an array of objects with at least { inputs }:

ls.test.each([
  { inputs: { query: "Q1" }, referenceOutputs: { answer: "A1" } },
  { inputs: { query: "Q2" }, referenceOutputs: { answer: "A2" } },
  // Optional additional fields: id, metadata, split
  { id: "custom-id", inputs: { query: "Q3" }, referenceOutputs: { answer: "A3" }, split: "hard" },
])(
  "case: ${inputs.query}",  // Name template — interpolates from row
  async ({ inputs, referenceOutputs, testMetadata }) => {
    // testMetadata.exampleId, testMetadata.datasetId, etc.
    const result = await runner.run({ query: inputs.query });
    // ...
  },
);

Reference: LangSmith integration

How datasets map

ConceptLangSmith entity
ls.describe(name, ...)Dataset (name = dataset name)
ls.test(name, { inputs, referenceOutputs }, fn)Example in dataset
Running the test suiteExperiment on the dataset
ls.logFeedback(...)Feedback on the experiment run
ls.logOutputs(...)Experiment output (called by runner)

Environment variables

VariablePurpose
EVAL_RUNNERWhich model runner to use (e.g. sonnet-4-5)
LANGSMITH_API_KEYLangSmith auth
LANGSMITH_PROJECTOverride tracing project (normally set via projectName)
LANGSMITH_TEST_TRACKINGSet to "false" to disable LangSmith reporting
ANTHROPIC_API_KEYFor Anthropic model runners
OPENAI_API_KEYFor OpenAI model runners

Available runners

Defined in internal/eval-harness/src/setup.ts:

Runner nameModel
sonnet-4-5Claude Sonnet 4.5
sonnet-4-5-thinkingClaude Sonnet 4.5 with extended thinking
opus-4-6Claude Opus 4.6
gpt-4.1GPT-4.1
gpt-4.1-miniGPT-4.1 Mini
o3-minio3-mini

Reference: Implementing published benchmarks

When implementing an existing benchmark, follow these attribution and methodology guidelines.

Attribution

Always credit the original benchmark authors. In the eval's README.md:

# <benchmark-name>

Implementation of [<Benchmark Name>](<paper-or-repo-url>) by <authors> (<year>).

> <brief description from the paper abstract>

## Citation

\`\`\`bibtex
@article{...}
\`\`\`

## Adaptations

<Describe any differences from the original benchmark methodology:>
- <e.g. "Subset of N cases selected for cost efficiency">
- <e.g. "Adapted for agentic tool-use evaluation rather than direct QA">
- <e.g. "Uses LLM-as-judge instead of human annotation">

Common benchmark patterns

QA / Factual benchmarks (e.g. MMLU, TrivialQA)

  • Test cases: question + gold answer
  • Scoring: exact match or fuzzy match on final text
  • Pattern: ls.test.each with fixture JSON

Multi-turn / Conversational (e.g. MT-Bench)

  • Test cases: conversation turns that build on each other
  • Scoring: LLM-as-judge on each turn
  • Pattern: sequential runner.run() calls sharing state, or multi-message queries

Tool-use benchmarks (e.g. ToolBench, API-Bank)

  • Test cases: task requiring specific tool calls
  • Scoring: trajectory matchers (correct tools called, correct args)
  • Pattern: runner.extend({ tools: [...] }) with custom tools that return canned responses

Code generation (e.g. HumanEval, SWE-bench)

  • Test cases: problem description + test suite
  • Scoring: execute generated code, check test results
  • Requires sandbox: yes — agent must run code and observe output
  • Pattern: runner.extend({ backend: sandbox }), write code to file, execute tests via sandbox

Memory / Long-context (e.g. oolong, needle-in-haystack)

  • Test cases: large context + retrieval question
  • Scoring: whether the agent finds the target information
  • Requires sandbox: no — seed files via initialFiles, check final text
  • Pattern: in-memory StateBackend is sufficient

Agentic benchmarks (e.g. AgentBench, WebArena)

  • Test cases: multi-step tasks requiring planning
  • Scoring: combination of trajectory analysis + final state checking
  • Requires sandbox: yes — agent needs shell access, package installs, environment interaction
  • Pattern: runner.extend({ backend: sandbox }) with per-test sandbox isolation

Handling large datasets

For benchmarks with thousands of cases:

  1. Subset selection — pick a representative subset (e.g. 100 cases per category). Document the selection criteria.
  2. Split support — use split field in test cases to categorise (e.g. "easy", "hard"). Run subsets via vitest filtering.
  3. Caching — download once, cache in .cache/ (gitignored).
  4. Cost awareness — estimate API cost before running. Log it in the README. Consider a --dry-run that validates fixtures without calling the LLM.

FAQ

What does the eval-writer skill do?

Create new eval suites for the deepagentsjs monorepo. Handles dataset design, test case scaffolding, scoring logic, vitest configuration, and LangSmith integration. Use when the user asks to: (1) create an eval, (2) write an evaluation, (3) add a benchmark, (4) build an eval suite, (5) evaluate agent behaviour, (6) add test cases for a capability, or (7) implement an existing benchmark (e.g. oolong, AgentBench, SWE-bench). Trigger on phrases like 'create eval', 'new eval', 'add eval', 'benchmark', 'evaluate', 'eval suite', 'write evals for'.

How do I install eval-writer?

Run npx skills add https://github.com/langchain-ai/deepagentsjs --skill eval-writer in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/eval-writer/ (personal) or .claude/skills/eval-writer/ (project).

Which AI agents can use eval-writer?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is eval-writer safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains eval-writer?

It is published by langchain-ai in langchain-ai/deepagentsjs, and it is an official skill from the maker of the technology it covers. It has 51 recorded installs.

More from langchain-ai/deepagentsjs

View all →

file-organizer

langchain-ai/deepagentsjs

Intelligently organizes your files and folders across your computer by understanding context, finding duplicates, suggesting better structures, and automating cleanup tasks. Reduces cognitive load and keeps your digital workspace tidy without manual effort.

OfficialAI & Agents74 installs

langsmith-trace

langchain-ai/deepagentsjs

INVOKE THIS SKILL when working with LangSmith tracing OR querying traces. Covers adding tracing to applications and querying/exporting trace data. Uses the langsmith CLI tool.

OfficialAI & Agents47 installs

langgraph-docs

langchain-ai/deepagentsjs

Use this skill for requests related to LangGraph in order to fetch relevant documentation to provide accurate, up-to-date guidance.

OfficialDocs & Writing38 installs

Popular ai & agents skills

View category →

google-agents-cli-publish

google/agents-cli

This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).

AI & Agents513.9K installs

google-agents-cli-eval

google/agents-cli

This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).

AI & Agents513.5K installs

skill-creator

anthropics/skills

Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.

OfficialAI & Agents416.4K installs

codex-pet

prime-skills/runcomfy-agent-skills

Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.

AI & Agents402.1K installs

full-output-enforcement

leonxlnx/taste-skill

Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.

AI & Agents401K installs

executing-plans

obra/superpowers

Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available

AI & Agents245.7K installs