langsmith-code-eval
Creates code-based evaluators for LangSmith-traced agents. Use when building custom evaluation logic, testing tool usage patterns, or scoring agent outputs programmatically. Triggers on requests to evaluate agents, create evaluators, or run experiments against LangSmith datasets.
- Installs
- 48
- Last 24h
- —
- Trending rank
- —
- Security
- Not audited
Install
Works with every agent the skills CLI supports.
npx skills add https://github.com/langchain-ai/lca-skills --skill langsmith-code-eval- 01
Install
Run the command in your project. The CLI asks which agents to install for.
- 02
Describe the task
Ask your agent for something this skill covers. It loads langsmith-code-eval automatically when relevant.
- 03
Review the result
The agent follows the SKILL.md steps, scripts and references bundled with the skill.
SKILL.md
The instructions your agent reads when the skill is triggered.
LangSmith Code Evaluator Creation
Creates evaluators for LangSmith experiments through structured inspection and implementation.
Prerequisites
langsmithPython package installedLANGSMITH_API_KEYenvironment variable set (check project's.envfile)
Workflow
Copy this checklist and track progress:
Evaluator Creation Progress:
- [ ] Step 1: Gather info from user
- [ ] Step 2: Inspect trace and dataset structure
- [ ] Step 3: Read agent code
- [ ] Step 4: Write evaluator
- [ ] Step 5: Write experiment runner
- [ ] Step 6: Run and iterate
Step 1: Gather Info from User
IMPORTANT: Do NOT search or explore the codebase. Ask the user all of these questions upfront using AskUserQuestion before doing anything else.
Ask the user the following in a single AskUserQuestion call:
- Python command: How do you run Python in this project? (e.g.,
python,python3,uv run python,poetry run python) - Agent file path: What is the path to your agent file?
- LangSmith project name: What is your LangSmith project name (where traces are logged)?
- LangSmith dataset name: What is the name of the dataset to evaluate against?
- Evaluation goal: What behavior should pass vs fail? Common types:
- Tool usage: Did the agent call the correct tool?
- Output correctness: Does output match expected format/content?
- Policy compliance: Did it follow specific rules?
- Classification: Did it categorize correctly?
Step 2: Inspect Trace and Dataset Structure
Using the info from Step 1, run the inspection scripts located in this skill's directory:
{python_cmd} {skill_dir}/scripts/inspect_trace.py PROJECT_NAME [RUN_ID]
{python_cmd} {skill_dir}/scripts/inspect_dataset.py DATASET_NAME
Replace {python_cmd} with the command from Step 1, and {skill_dir} with this skill's directory path.
Verify the trace matches the agent:
- Does the trace type match? (e.g., OpenAI trace for OpenAI agent)
- Does it contain the data needed for evaluation?
- If mismatched, clarify before proceeding.
From the dataset inspection, note:
- Input schema (what gets passed to the agent)
- Output schema (reference/expected outputs)
- Metadata fields (e.g.,
expected_tool,difficulty, labels)
The dataset metadata often contains ground truth for evaluation (e.g., which tool should be called, expected classification).
Step 3: Read Agent Code
Read the agent file provided in Step 1 to identify:
- Entry point function (look for
@traceabledecorator) - Available tools
- Output format (what the function returns)
Step 4: Write the Evaluator
Create evaluator functions based on trace and dataset structure. See EVALUATOR_REFERENCE.md for function signatures and return formats.
Step 5: Write Experiment Runner
Create a script that:
- Imports the agent's entry function
- Wraps it as a target function
- Runs
evaluate()oraevaluate()against the dataset
See EVALUATOR_REFERENCE.md for evaluate() usage.
Step 6: Run and Iterate
Execute the experiment, review results in LangSmith, refine evaluators as needed.
FAQ
What does the langsmith-code-eval skill do?
Creates code-based evaluators for LangSmith-traced agents. Use when building custom evaluation logic, testing tool usage patterns, or scoring agent outputs programmatically. Triggers on requests to evaluate agents, create evaluators, or run experiments against LangSmith datasets.
How do I install langsmith-code-eval?
Run npx skills add https://github.com/langchain-ai/lca-skills --skill langsmith-code-eval in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/langsmith-code-eval/ (personal) or .claude/skills/langsmith-code-eval/ (project).
Which AI agents can use langsmith-code-eval?
Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.
Is langsmith-code-eval safe to install?
No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.
Who maintains langsmith-code-eval?
It is published by langchain-ai in langchain-ai/lca-skills, and it is an official skill from the maker of the technology it covers. It has 48 recorded installs.
More from langchain-ai/lca-skills
langsmith-fetch
langchain-ai/lca-skills
Fetches LangSmith traces for debugging agent behavior. Use when troubleshooting agent issues, reviewing conversation history, or investigating tool calls.
Popular ai & agents skills
google-agents-cli-publish
google/agents-cli
This skill should be used when the user wants to "publish an agent", "publish my ADK agent", "register an agent with Gemini", "publish to Gemini" (formerly Gemini Enterprise), or needs guidance on the agents-cli publish gemini command. Also use when the user wants to "manage agents in Agent Registry", "list/update/delete registered agents", or "register an MCP server". Covers ADK vs A2A registration modes, programmatic and interactive usage, flag reference, auto-detection from deployment metadata, Agent Registry fleet management, and troubleshooting. Part of the agents-cli skills suite. Do NOT use for deployment (use google-agents-cli-deploy).
google-agents-cli-eval
google/agents-cli
This skill should be used when the user wants to "run an evaluation", "evaluate my agent", "evaluate my ADK agent", "write an eval dataset", "analyze eval failures", "compare eval results", "optimize agent", or needs guidance on the Gemini platform eval methodology and the Quality Flywheel. Covers eval metrics, dataset schema, LLM-as-judge scoring, and common failure causes. Applies to any agents-cli project, whatever framework the agent is written in. Do NOT use for agent API code patterns (ADK: use google-agents-cli-adk-code), deployment (use google-agents-cli-deploy), or project scaffolding (use google-agents-cli-scaffold).
skill-creator
anthropics/skills
Create new skills, modify and improve existing skills, and measure skill performance. Use when users want to create a skill from scratch, edit, or optimize an existing skill, run evals to test a skill, benchmark skill performance with variance analysis, or optimize a skill's description for better triggering accuracy.
codex-pet
prime-skills/runcomfy-agent-skills
Codex Pet generator on RunComfy. Build a Codex-compatible Codex Pet spritesheet.webp + pet.json from a single reference image, drop it into ${CODEXHOME:-$HOME/.codex}/pets/ / and Codex picks it up as a custom Codex Pet next to the 8 built-ins. This skill produces the exact Codex Pet atlas Codex expects (1536x1872 PNG/WebP, 8 cols x 9 rows, 192x208 cells, 9 animation states — idle, running-right, running-left, waving, jumping, failed, waiting, running, review). Calls OpenAI GPT Image 2 edit ONCE via the local RunComfy CLI as runcomfy run openai/gpt-image-2/edit to produce a canonical Codex Pet pose, then assembles all 9 animation rows programmatically with ImageMagick micro-transforms — no Codex Pro, no $imagegen, no OPENAIAPIKEY required, only RUNCOMFYTOKEN. Triggers on "codex pet", "create codex pet", "make codex pet", "hatch codex pet", "/hatch image", "desktop pet codex", "codex pets", "spritesheet.webp", or any explicit ask to build a custom pet for OpenAI Codex.
full-output-enforcement
leonxlnx/taste-skill
Overrides default LLM truncation behavior. Enforces complete code generation, bans placeholder patterns, and handles token-limit splits cleanly. Apply to any task requiring exhaustive, unabridged output.
executing-plans
obra/superpowers
Use when executing an implementation plan in the current session as the implementer yourself — your human partner chose inline execution, or no subagent tool is available