MCPInstall

langsmith-dataset

by langchain-ai OfficialData & Databases

INVOKE THIS SKILL when creating evaluation datasets, uploading datasets to LangSmith, or managing existing datasets. Covers dataset types (finalresponse, singlestep, trajectory, RAG), CLI management commands, SDK-based creation, and example management. Uses the langsmith CLI tool.

Installs
55
Last 24h
—
Trending rank
—
Security
Not audited

Install

Works with every agent the skills CLI supports.

npx skills add https://github.com/langchain-ai/skills-benchmarks --skill langsmith-dataset
  1. 01

    Install

    Run the command in your project. The CLI asks which agents to install for.

  2. 02

    Describe the task

    Ask your agent for something this skill covers. It loads langsmith-dataset automatically when relevant.

  3. 03

    Review the result

    The agent follows the SKILL.md steps, scripts and references bundled with the skill.

SKILL.md

The instructions your agent reads when the skill is triggered.

<oneliner> Create, manage, and upload evaluation datasets to LangSmith for testing and validation. </oneliner> <setup> Environment Variables 
LANGSMITH_API_KEY=lsv2_pt_your_api_key_here          # REQUIRED
LANGSMITH_PROJECT=your-project-name                   # Check this to know which project has traces
LANGSMITH_WORKSPACE_ID=your-workspace-id              # Optional: for org-scoped keys

Authentication is REQUIRED: either set the LANGSMITH_API_KEY environment variable, or pass the --api-key flag to CLI commands (preferred):

langsmith dataset list --api-key $LANGSMITH_API_KEY

IMPORTANT: Always check the environment variables or .env file for LANGSMITH_PROJECT before querying or interacting with LangSmith. This tells you which project contains the relevant traces and data. If the LangSmith project is not available, use your best judgement to identify the right one.

Python Dependencies

pip install langsmith

JavaScript Dependencies

npm install langsmith

CLI Tool

curl -sSL https://raw.githubusercontent.com/langchain-ai/langsmith-cli/main/scripts/install.sh | sh
</setup> <usage> Use the `langsmith` CLI to manage datasets and examples.

Dataset Commands

  • langsmith dataset list - List datasets in LangSmith
  • langsmith dataset get <name-or-id> - View dataset details
  • langsmith dataset create --name <name> - Create a new empty dataset
  • langsmith dataset delete <name-or-id> - Delete a dataset
  • langsmith dataset export <name-or-id> <output-file> - Export dataset to local JSON file
  • langsmith dataset upload <file> --name <name> - Upload a local JSON file as a dataset

Example Commands

  • langsmith example list --dataset <name> - List examples in a dataset
  • langsmith example create --dataset <name> --inputs <json> - Add an example to a dataset
  • langsmith example delete <example-id> - Delete an example

Experiment Commands

  • langsmith experiment list --dataset <name> - List experiments for a dataset
  • langsmith experiment get <name> - View experiment results

Common Flags

  • --limit N - Limit number of results
  • --yes - Skip confirmation prompts (use with caution)

IMPORTANT - Safety Prompts:

  • The CLI prompts for confirmation before destructive operations (delete, overwrite)
  • If you are running with user input: ALWAYS wait for user input; NEVER use --yes unless the user explicitly requests it
  • If you are running non-interactively: Use --yes to skip confirmation prompts </usage>

<dataset_types_overview> Common evaluation dataset types:

  • final_response - Full conversation with expected output. Tests complete agent behavior.
  • single_step - Single node inputs/outputs. Tests specific node behavior (e.g., one LLM call or tool).
  • trajectory - Tool call sequence. Tests execution path (ordered list of tool names).
  • rag - Question/chunks/answer/citations. Tests retrieval quality. </dataset_types_overview>

<creating_datasets>

Creating Datasets

Datasets are JSON files with an array of examples. Each example has inputs and outputs.

From Exported Traces (Programmatic)

Export traces first, then process them into dataset format using code:

# 1. Export traces to JSONL files
langsmith trace export ./traces --project my-project --limit 20 --full --api-key $LANGSMITH_API_KEY
<python> ```python import json from pathlib import Path from langsmith import Client

client = Client()

2. Process traces into dataset examples

examples = [] for jsonl_file in Path("./traces").glob("*.jsonl"): runs = [json.loads(line) for line in jsonl_file.read_text().strip().split("\n")] root = next((r for r in runs if r.get("parent_run_id") is None), None) if root and root.get("inputs") and root.get("outputs"): examples.append({ "trace_id": root.get("trace_id"), "inputs": root["inputs"], "outputs": root["outputs"] })

3. Save locally

with open("/tmp/dataset.json", "w") as f: json.dump(examples, f, indent=2)

</python>

<typescript>
```typescript
import { Client } from "langsmith";
import { readFileSync, writeFileSync, readdirSync } from "fs";
import { join } from "path";

const client = new Client();

// 2. Process traces into dataset examples
const examples: Array<{trace_id?: string, inputs: Record<string, any>, outputs: Record<string, any>}> = [];
const files = readdirSync("./traces").filter(f => f.endsWith(".jsonl"));

for (const file of files) {
  const lines = readFileSync(join("./traces", file), "utf-8").trim().split("\n");
  const runs = lines.map(line => JSON.parse(line));
  const root = runs.find(r => r.parent_run_id == null);
  if (root?.inputs && root?.outputs) {
    examples.push({ trace_id: root.trace_id, inputs: root.inputs, outputs: root.outputs });
  }
}

// 3. Save locally
writeFileSync("/tmp/dataset.json", JSON.stringify(examples, null, 2));
</typescript>

Upload to LangSmith

# Upload local JSON file as a dataset
langsmith dataset upload /tmp/dataset.json --name "My Evaluation Dataset" --api-key $LANGSMITH_API_KEY

Using the SDK Directly

<python> ```python from langsmith import Client

client = Client()

Create dataset and add examples in one step

dataset = client.create_dataset("My Dataset", description="Evaluation dataset")

client.create_examples( inputs=[{"query": "What is AI?"}, {"query": "Explain RAG"}], outputs=[{"answer": "AI is..."}, {"answer": "RAG is..."}], dataset_name="My Dataset", )

</python>

<typescript>
```typescript
import { Client } from "langsmith";

const client = new Client();

// Create dataset and add examples
const dataset = await client.createDataset("My Dataset", {
  description: "Evaluation dataset",
});

await client.createExamples({
  inputs: [{ query: "What is AI?" }, { query: "Explain RAG" }],
  outputs: [{ answer: "AI is..." }, { answer: "RAG is..." }],
  datasetName: "My Dataset",
});
</typescript> </creating_datasets>

<dataset_structures>

Dataset Structures by Type

Final Response

{"trace_id": "...", "inputs": {"query": "What are the top genres?"}, "outputs": {"response": "The top genres are..."}}

Single Step

{"trace_id": "...", "inputs": {"messages": [...]}, "outputs": {"content": "..."}, "metadata": {"node_name": "model"}}

Trajectory

{"trace_id": "...", "inputs": {"query": "..."}, "outputs": {"expected_trajectory": ["tool_a", "tool_b", "tool_c"]}}

RAG

{"trace_id": "...", "inputs": {"question": "How do I..."}, "outputs": {"answer": "...", "retrieved_chunks": ["..."], "cited_chunks": ["..."]}}

</dataset_structures>

<script_usage>

CLI Usage

# List all datasets
langsmith dataset list --api-key $LANGSMITH_API_KEY

# Get dataset details
langsmith dataset get "My Dataset" --api-key $LANGSMITH_API_KEY

# Create an empty dataset
langsmith dataset create --name "New Dataset" --description "For evaluation" --api-key $LANGSMITH_API_KEY

# Upload a local JSON file
langsmith dataset upload /tmp/dataset.json --name "My Dataset" --api-key $LANGSMITH_API_KEY

# Export a dataset to local file
langsmith dataset export "My Dataset" /tmp/exported.json --limit 100 --api-key $LANGSMITH_API_KEY

# Delete a dataset
langsmith dataset delete "My Dataset" --api-key $LANGSMITH_API_KEY

# List examples in a dataset
langsmith example list --dataset "My Dataset" --limit 10 --api-key $LANGSMITH_API_KEY

# Add an example
langsmith example create --dataset "My Dataset" \
  --inputs '{"query": "test"}' \
  --outputs '{"answer": "result"}' --api-key $LANGSMITH_API_KEY

# List experiments
langsmith experiment list --dataset "My Dataset" --api-key $LANGSMITH_API_KEY
langsmith experiment get "eval-v1" --api-key $LANGSMITH_API_KEY

</script_usage>

<example_workflow> Complete workflow from traces to uploaded LangSmith dataset:

# 1. Export traces from LangSmith
langsmith trace export ./traces --project my-project --limit 20 --full --api-key $LANGSMITH_API_KEY

# 2. Process traces into dataset format (using Python/JS code)
# See "Creating Datasets" section above

# 3. Upload to LangSmith
langsmith dataset upload /tmp/final_response.json --name "Skills: Final Response" --api-key $LANGSMITH_API_KEY
langsmith dataset upload /tmp/trajectory.json --name "Skills: Trajectory" --api-key $LANGSMITH_API_KEY

# 4. Verify upload
langsmith dataset list --api-key $LANGSMITH_API_KEY
langsmith dataset get "Skills: Final Response" --api-key $LANGSMITH_API_KEY
langsmith example list --dataset "Skills: Final Response" --limit 3 --api-key $LANGSMITH_API_KEY

# 5. Run experiments
langsmith experiment list --dataset "Skills: Final Response" --api-key $LANGSMITH_API_KEY

</example_workflow>

<troubleshooting> **Dataset upload fails:** - Verify LANGSMITH_API_KEY is set - Check JSON file is valid: each element needs `inputs` (and optionally `outputs`) - Dataset name must be unique, or delete existing first with `langsmith dataset delete`

Empty dataset after upload:

  • Verify JSON file contains an array of objects with inputs key
  • Check file isn't empty: langsmith example list --dataset "Name"

Export has no data:

  • Ensure traces were exported with --full flag to include inputs/outputs
  • Verify traces have both inputs and outputs populated

Example count mismatch:

  • Use langsmith dataset get "Name" to check remote count
  • Compare with local file to verify upload completeness </troubleshooting>
</output>

FAQ

What does the langsmith-dataset skill do?

INVOKE THIS SKILL when creating evaluation datasets, uploading datasets to LangSmith, or managing existing datasets. Covers dataset types (finalresponse, singlestep, trajectory, RAG), CLI management commands, SDK-based creation, and example management. Uses the langsmith CLI tool.

How do I install langsmith-dataset?

Run npx skills add https://github.com/langchain-ai/skills-benchmarks --skill langsmith-dataset in your project. The skills CLI detects the agents you use and copies the skill into the right folder. For Claude Code you can also place the skill folder at ~/.claude/skills/langsmith-dataset/ (personal) or .claude/skills/langsmith-dataset/ (project).

Which AI agents can use langsmith-dataset?

Any agent that supports the open Agent Skills format (a folder with a SKILL.md file), including Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, Windsurf, Cline, OpenCode and more.

Is langsmith-dataset safe to install?

No partner security audit is available yet. Skills can run commands and scripts, so read the SKILL.md and any bundled files before installing.

Who maintains langsmith-dataset?

It is published by langchain-ai in langchain-ai/skills-benchmarks, and it is an official skill from the maker of the technology it covers. It has 55 recorded installs.

More from langchain-ai/skills-benchmarks

View all →

react-components

langchain-ai/skills-benchmarks

Modern React component patterns with hooks and TypeScript

OfficialFrontend & Design85 installs

langsmith-trace

langchain-ai/skills-benchmarks

INVOKE THIS SKILL when working with LangSmith tracing OR querying traces. Covers adding tracing to applications and querying/exporting trace data. Uses the langsmith CLI tool.

OfficialAI & Agents64 installs

langgraph-fundamentals

langchain-ai/skills-benchmarks

Open the skill to read its SKILL.md instructions.

OfficialAI & Agents64 installs

langchain-oss-primer

langchain-ai/skills-benchmarks

Open the skill to read its SKILL.md instructions.

OfficialAI & Agents62 installs

langchain-dependencies

langchain-ai/skills-benchmarks

Open the skill to read its SKILL.md instructions.

OfficialAI & Agents61 installs

framework-selection

langchain-ai/skills-benchmarks

Open the skill to read its SKILL.md instructions.

OfficialAI & Agents60 installs

Popular data & databases skills

View category →

supabase-postgres-best-practices

supabase/agent-skills

Postgres best practices maintained by Supabase, for Postgres running anywhere. Load this skill BEFORE writing or changing anything that lives in a Postgres database: creating or altering tables and columns (including choosing column types), schema design, migrations and declarative schema files, RLS policies and the tests that verify them, indexes, triggers, database functions, queues and scheduled jobs (pgcron, pgmq), vector/semantic search (pgvector), and restoring dumps (pgrestore) or importing data. Also load it when diagnosing slow queries, high CPU, timeouts, EXPLAIN plans, connection exhaustion, locking, bloat, or rows visible to the wrong user or tenant. This is not just a performance guide — schema, migration, security, and SQL authoring tasks need these rules too, even for a one-column change or a single query.

OfficialData & Databases462.2K installs

prisma-client-api

prisma/skills

Prisma Client API reference covering model queries, filters, operators, and client methods. Use when writing database queries, using CRUD operations, filtering data, or configuring Prisma Client. Triggers on "prisma query", "findMany", "create", "update", "delete", "$transaction".

OfficialData & Databases393.5K installs

prisma-database-setup

prisma/skills

Deprecated compatibility name for prisma-orm-setup. Use only when a user or existing integration explicitly invokes prisma-database-setup; otherwise select prisma-orm-setup.

OfficialData & Databases392.3K installs

prisma-cli

prisma/skills

Prisma ORM 7 CLI reference for existing Prisma 7 applications or explicitly selected Prisma 7 workflows. Covers prisma init, generate, migrate, db, dev, studio, and mcp. For setup, connection repair, or Prisma 8, use prisma-orm-setup; not the Prisma Platform CLI.

OfficialData & Databases390.8K installs

prisma-postgres

prisma/skills

Deprecated compatibility name for prisma-postgres-setup. Use only when a user or existing integration explicitly invokes prisma-postgres; otherwise select prisma-postgres-setup.

OfficialData & Databases384.4K installs

prisma-driver-adapter-implementation

prisma/skills

Required reference for Prisma ORM 7 SQL driver adapter work. Use when implementing or modifying adapters, adding database drivers, or touching SqlDriverAdapter, Transaction, savepoint, result mapping, or DriverAdapterError behavior. Covers current transaction lifecycle, optional savepoint hooks, original database-error preservation, and verification.

OfficialData & Databases378.9K installs