rewardkit

Help the user write task verifiers with Reward Kit. Reward Kit is a lightweight Python package that turns a directory of criteria files into a reward score. Each criterion is a Python function call or a TOML judge file; folders become separate rewards.

Setup in a Harbor task

Put criteria alongside test.sh in the task's tests/ directory:

tests/
├── test.sh
├── checks.py         # programmatic criteria
└── judge.toml        # optional LLM/agent judge

tests/test.sh:

#!/bin/bash
uvx --python '>=3.12,<3.14' --from 'harbor-rewardkit==0.2.*' rewardkit /tests

This runs all criteria in /tests/ against the workspace at /app and writes /logs/verifier/reward.json. Defaults match Harbor's conventions — no extra config needed.

If judge criteria need API keys, pass them through task.toml:

[verifier.env]
ANTHROPIC_API_KEY = "${ANTHROPIC_API_KEY}"

Ask whether Reward Kit should run in the agent's shared environment or in a separate verifier environment. Prefer a separate verifier environment when judge prompts, grading dependencies, API keys, or clean-room checks should not be available to the agent:

[environment]
network_mode = "no-network"   # Agent env baseline — offline during agent.run()

[verifier]
environment_mode = "separate"

[verifier.environment]
network_mode = "public"     # Verifier env baseline — LLM judge API calls
docker_image = "python:3.12-slim"

In shared mode, the verifier runs in the agent container and inherits [environment].network_mode. Put [verifier].network_mode only when verify() needs different network access than the agent phase (a phase override, not a baseline). If agent and verifier need different baselines without runtime switching, use environment_mode = "separate" and set [verifier.environment].network_mode.

Judge criteria that call external APIs need a public baseline or allowlist on the verifier environment. Programmatic checks that only read local files can use no-network.

In separate mode, tests/ is the verifier image build context and must provide /tests/test.sh at runtime; Harbor does not upload tests/ into the running verifier container.

Programmatic criteria

Call built-ins from any .py file in tests/:

import rewardkit as rk

rk.file_exists("output.txt")
rk.file_contains("output.txt", "hello")
rk.command_succeeds("python main.py", weight=2.0)
rk.json_key_equals("result.json", "status", "ok")

All criteria accept weight (default 1.0) and isolated (default False, runs in overlayfs so side effects don't leak).

Available built-ins

• Files: file_exists, file_not_exists, file_contains, file_contains_regex, file_matches, files_equal, diff_ratio
• Commands: command_succeeds, command_output_contains, command_output_matches, command_output_matches_regex (30s default timeout, optional cwd)
• Data: json_key_equals, json_path_equals, csv_cell_equals, xlsx_cell_equals (needs [office] extra), sqlite_query_equals
• HTTP: http_status_equals, http_response_contains
• Images: image_similarity, image_size_equals (needs [image] extra)
• Trajectory: trajectory_tool_used, trajectory_tool_not_used, trajectory_turn_count

For extras, install with uv tool install harbor-rewardkit[all].

Custom criteria

Use the @criterion decorator. First parameter is always workspace: Path. Returns bool, float, or a dict with score plus optional reasoning, confidence, and model:

from pathlib import Path
from rewardkit import criterion

@criterion
def has_valid_output(workspace: Path) -> bool:
    return (workspace / "output.txt").read_text().strip() != ""

Zero-parameter criteria auto-register. Criteria with extra args must be called via rk:

@criterion(description="output has at least {n} lines")
def has_n_lines(workspace: Path, n: int) -> bool:
    return len((workspace / "output.txt").read_text().splitlines()) >= n

rk.has_n_lines(10, weight=2.0)
rk.has_n_lines(50, weight=1.0)

For criteria shared across reward subdirs, define with shared=True in a root-level file and call from subdirs.

Judge criteria (LLM or agent-as-a-judge)

For subjective checks (quality, readability, edge cases), create a TOML file:

[judge]
judge = "anthropic/claude-opus-5-5"   # LiteLLM model string
files = ["/app/main.py"]

[[criterion]]
description = "Is the code correct?"
type = "binary"

[[criterion]]
description = "How readable is the code?"
type = "likert"
points = 5
weight = 2.0

Criterion types:

• binary — yes/no → 1.0 or 0.0
• likert — 1..points, normalized to [0, 1]
• numeric — min..max, normalized to [0, 1]
• rubric — 2 to 10 described levels forming a scale from worst to best; position sets the score

Agent judges

Agent judges shell out to a CLI and can explore the filesystem:

[judge]
judge = "claude-code"
model = "anthropic/claude-opus-5-5"
isolated = true

[[criterion]]
description = "Does the solution handle edge cases?"
type = "binary"

Slower and more expensive than LLM judges, but they can run commands and inspect files.

JEV judge

JEV is a new type of language model from TypeSafe. It answers each criterion with a probability or a rubric score and returns no reasoning, so it is fast and cheap. It needs the jev extra (harbor-rewardkit[jev]) and TYPESAFE_API_KEY.

[judge]
judge = "jev"
files = ["/app/answer.md"]

[[criterion]]
description = "Does the answer address the requested task?"

[[criterion]]
description = "How complete is the answer?"
type = "rubric"
levels = ["Omits the information", "Covers part of it", "Covers all of it"]

Binary criteria pass at a probability of 0.5 or higher. Only binary and rubric criteria and text files are supported; atif-trajectory and prompt_template are not. The task image needs CA certificates (ca-certificates on Debian and Ubuntu), because the TypeSafe SDK verifies TLS against the system trust store.

Useful [judge] options

timeout (default 300), reasoning_effort (low|medium|high for every LLM and agent judge, other levels depend on the judge; jev accepts none), reference (path to reference solution), atif-trajectory (evaluate the agent's trajectory), weight, prompt_template (custom prompt with {criteria} placeholder), samples (run the judge N times and score each criterion with its median sample), guard (protects the judge from instructions the agent wrote into the graded files; "flag" reports them under guard in the details, "penalize" also scores a flagged submission 0), isolated (agent judges only: read-only overlayfs copy of the workspace). More than one agent run in a task (several agent judges, samples above 1, or individual mode with several criteria) requires isolated = true on each agent judge, which needs a separate verifier with fuse-overlayfs and mount privileges; --yolo skips the check.

Scoring aggregation (within one judge TOML)

[scoring]
aggregation = "all-pass"   # weighted-mean | weighted-sum | all-pass | any-pass | threshold | required-pass
threshold = 0.7             # only for threshold

all-pass, any-pass, and required-pass count a score as passing only when it is 1.0, so a Likert or numeric criterion needs full marks.

Only affects how this file's own criteria combine. To aggregate across dimensions, see Aggregating dimensions.

Scoring config for programmatic files

Each .py file that registers criteria is an equal-weighted scoring component named after its filename stem. Files that only provide imports or shared criterion factories and register no checks are ignored. To change how criteria within a file combine, use [scoring.<stem>] in the same directory's reward.toml:

# tests/structure/reward.toml
[scoring.files_exist]       # configures files_exist.py
aggregation = "all-pass"

[scoring.behavior]          # configures behavior.py
aggregation = "threshold"
threshold = 0.75

Each entry takes the same aggregation values as a judge TOML. Unknown keys and stems that do not resolve to a criterion-bearing Python file raise.

Directories may be nested recursively. A non-root directory can aggregate its local Python files, local judges, and immediate child directories with one unnamed [[reward]] table:

# tests/correctness/reward.toml
[[reward]]
aggregation = "weighted-mean"
weights = { files = 2.0, behavior = 1.0 }

Membership is implicit. Child directories have weight 1.0 unless overridden; use filename stems for local Python files and judge TOMLs, and directory names for child groups. Without [[reward]], the directory defaults to weighted mean.

Multi-reward tasks

Put criteria in subdirectories — each becomes a separate reward:

tests/
├── test.sh
├── correctness/
│   └── check.py
├── structure/
│   └── files_exist.py
└── quality/
    └── quality.toml

Judge TOMLs may also sit directly at the tests root alongside reward subdirectories. Each is exposed as a top-level reward named after its filename stem and can be referenced by a root aggregation.

Criterion-bearing Python files at the tests root are also top-level dimensions named after their stems. Root support files that register no criteria are ignored.

Produces:

{ "correctness": 0.75, "structure": 1.0, "quality": 0.6 }

Aggregating dimensions

To add aggregated scores on top of the per-dimension keys, add a root-level tests/reward.toml with one or more [[reward]] tables. Each adds one key to reward.json, aggregating the dimensions with the same modes as [scoring]:

# tests/reward.toml
[[reward]]
name = "reward"
aggregation = "weighted-mean"
weights = { correctness = 2.0, quality = 1.0 }

{ "correctness": 0.75, "structure": 1.0, "quality": 0.6, "reward": 0.775 }

The per-dimension scores stay; aggregated keys are added alongside them (a name may not collide with a dimension). Top-level dimensions have equal weight unless that aggregation's inline map overrides them; reward-details.json keeps the full recursive breakdown.

Output files

• /logs/verifier/reward.json — per-reward scores
• /logs/verifier/reward-details.json — per-criterion results, judge reasoning, errors

Multi-step tasks

In a multi-step task, each step has its own tests/ under steps/{name}/tests/, and the verifier runs once per step. Reward Kit behaves the same as in a single-step task: for each step it reads /tests, runs the criteria against /app, and writes /logs/verifier/reward.json for that step. Harbor then aggregates per-step results into a trial-level reward via multi_step_reward_strategy in task.toml — aggregation happens outside Reward Kit, so don't try to encode cross-step logic in your criteria.

A task-level tests/ directory (at the task root) is uploaded to /tests first, then the step's own tests/ is layered on top (same-name files win). Put shared helpers (common checks.py functions with shared=True, fixture files, a fallback test.sh) at the task level, and step-specific criteria under each step.

Multi-reward subdirectories still work within a step: steps/foo/tests/ can contain correctness/, structure/, quality/ — each produces a separate reward key for that step, and multi_step_reward_strategy = "mean" averages each key across steps. Use "final" when the last step is an end-to-end check whose rewards already represent the full task.

When to reach for what

• Use built-ins for file existence, string matches, command output, JSON/CSV checks, HTTP probes.
• Use @criterion when logic is task-specific but still programmatic.
• Use LLM judges for subjective quality dimensions (readability, correctness of prose).
• Use agent judges when the rubric requires exploring the filesystem or running code (e.g. "does the test suite actually pass?").
• Use subdirectories when you want separate scores (correctness vs structure vs quality) rather than one blended number.
• Use isolated=True for any criterion that runs mutating commands, so it doesn't corrupt the workspace for other criteria.

Working example

See examples/tasks/reward-kit-example/ in the Harbor repo.