Phoenix for Harbor

Use the Phoenix Harbor plugin to record Harbor agent evaluations as versioned Phoenix datasets, experiments, runs, scores, and ATIF traces.

Harbor runs agents and verifiers. Phoenix records and compares their results. Do not describe Phoenix as executing Harbor tasks or recalculating Harbor rewards.

Requirements

• Python 3.12 or newer
• Harbor 0.21.0 or newer
• arize-phoenix-client installed with the harbor extra
• Phoenix server 15.0 or newer

Install the client and Harbor in the same Python environment:

uv pip install "arize-phoenix-client[harbor]"

Choose the trace mode

Use atif unless the agent has no ATIF trajectory or the user does not want traces. ATIF is the default. It reads trajectory files after the final trial attempt, so the sandbox needs no Phoenix endpoint, credentials, instrumentation, or outbound network access.

Use null to record datasets, experiments, runs, and evaluations without traces:

--plugin-kwarg trace_mode=null

Live OpenTelemetry Protocol (OTLP) support is deferred to a follow-up. This release accepts atif or null, and does not link live OpenTelemetry agent traces to experiment runs.

Set the Phoenix destination

Set the endpoint for the Phoenix plugin process. ATIF agents and their sandboxes do not connect to Phoenix. Prefer environment variables so credentials do not enter shell history or job configuration:

export PHOENIX_COLLECTOR_ENDPOINT=http://localhost:6006
export PHOENIX_API_KEY=your-api-key

Omit PHOENIX_API_KEY when the Phoenix instance does not require authentication. The endpoint and api_key plugin kwargs override these values when the user asks for per-job settings.

Add the plugin

Add --plugin arize-phoenix to the user's existing harbor run command. Preserve their dataset, agent, model, environment, concurrency, retry, and task selections.

harbor run \
  -d terminal-bench/terminal-bench-2 \
  -a terminus-2 \
  -m openai/gpt-5-mini \
  --plugin arize-phoenix \
  --yes

Do not invent or replace Harbor settings that are unrelated to Phoenix.

Predict the Phoenix records

Use this mapping when explaining a job or checking its results:

Each single-step or multi-step Harbor task becomes one Phoenix dataset example. A multi-step example input includes its ordered step names and instructions. Phoenix examples keep output empty unless the task declares a reference file.

The plugin records only the terminal physical attempt for a logical trial. An attempt that Harbor will retry does not create a Phoenix run. Completion order does not define repetition numbers.

When a saved terminal ATIF trajectory ends with a user-facing textual agent turn, the plugin records it in chat-message format so Phoenix experiment comparisons render it as Markdown. Structured messages contribute their text parts in order; media parts are omitted. The output stays empty for missing or invalid trajectories, terminal tool calls, media-only turns, and state-only tasks. Multi-step tasks use the last attempted step, and continued trajectories use the terminal continuation. Output extraction still runs with trace_mode=null; that setting disables trace creation, not result display.

Successful runs written by older plugin versions keep their legacy Harbor metadata output because Phoenix runs are immutable. Resume recognizes that exact legacy shape and reuses the run.

Add optional reference outputs

A task can declare a checked-in reference file in its root task.toml:

[metadata.arize-phoenix]
reference_output_path = "tests/expected.json"

The plugin reads this UTF-8 JSON file during setup. A JSON string becomes {"messages": [{"role": "assistant", "content": "the reference text"}]}. A JSON object is stored unchanged as the example output. Other top-level types must be wrapped in an object. No setting means an empty output, even if an expected.json file exists.

Paths are relative to the downloaded task root on the Harbor host. Absolute paths, .. components, and symlinks escaping that root are rejected. Missing, unreadable, or invalid configured files fail setup before trials run. Keep references beside verifier assets, out of the agent workspace.

This works for local datasets, direct tasks, and published tasks. Multi-step tasks have one reference for the whole task, usually its expected final result. Reference-content changes create a dataset version; existing experiments keep their original version. References do not change Harbor's grading.

Name the dataset

The plugin infers a Phoenix dataset name for each supported single-source job. Provide dataset=<name> only when a job contains several direct tasks, which have no shared collection name, or when you want to customize the dataset's display name in Phoenix.

The inferred names are:

To name several direct tasks or override an inferred name, add this setting to the Harbor command:

--plugin-kwarg dataset=release-candidate-tasks

Stop and explain the constraint if the job has any unsupported source shape:

• more than one configured dataset;
• both a configured dataset and direct tasks;
• several direct tasks without dataset=<name>;
• duplicate task IDs; or
• a regrade job or another job derived from previous Harbor results.

The plugin synchronizes the complete resolved task set at job start. An unchanged set reuses the dataset version. A task addition, removal, or content change creates a version. Existing experiments stay pinned to their creation-time version.

Name experiments

The default template is:

{job.name} · {agent.name} · {agent.model}

For one agent configuration, an exact name is valid:

--plugin-kwarg experiment_name=release-candidate

For several agent configurations, use experiment_name_template. Available fields are:

• {job.name}
• {job.id}
• {dataset.name}
• {agent.name}
• {agent.model}
• {agent.short_digest}

Agent names do not need to be unique. Two agents with the same name but different effective configurations each get an experiment. If rendered names collide, the plugin appends the short agent digest. Stable identity comes from the Harbor job ID and effective agent configuration, not the display name.

Interpret scores

Keep behavioral outcomes separate from execution health. Phoenix does not run another evaluator. The plugin records Harbor's completed verifier rewards as named experiment evaluations with the CODE annotator kind.

Do not infer reward from another lone key. Check its coverage before computing cross-task summaries.

A run may contain rewards and still have infra_ok=0. Harbor can produce verifier output before or alongside a step exception. Preserve both facts when explaining the result.

For a multi-step task, trial-level reward evaluations include multi_step_reward_strategy metadata. Harbor's omitted default resolves to mean; preserve an explicit final. Step evaluations and infra_ok do not include this field.

For comparisons:

1. Check the fraction of runs with reward.
2. Compare reward among behaviorally completed runs.
3. Compare infra_ok to find environment, timeout, agent-process, or verifier reliability problems.
4. Use step scores and the linked trace to locate the failure within a task.

Interpret ATIF traces

One logical trial maps to one trace and one Phoenix session. The trace starts with a plugin-owned harbor.trial CHAIN span. Multi-step trials add one harbor.step span per attempted step:

harbor.trial <task>                  CHAIN
  harbor.step 1 <step name>          CHAIN, multi-step trials only
    <agent>                          AGENT
      turn 1                         AGENT, multi-turn trajectories only
        iteration 1                  CHAIN
          <model>                    LLM
          <tool>                     TOOL
            <subagent>               AGENT

Single-step trajectories attach directly to the trial root. Each multi-step harbor.step span carries its instruction, timing, exception status, and any verifier rewards. A step remains visible even when its trajectory is missing. All step spans and trajectories share the trial root.

Agent, model, and tool spans use their ATIF names. Fresh agent operations use iteration N; context-management operations use compaction N; and other operational system steps use system event N. Multi-turn trajectories add turn N spans. Steps with llm_call_count: 0 keep their operation and tool spans but do not create an LLM span. Continuation roots use <agent> (continuation N).

The converter supports ATIF v1.0 through v1.7. It reconstructs LLM inputs from ATIF and marks them with metadata.atif.input_source = "reconstructed". Copied prompt history contributes to those inputs without creating spans. It pairs an observation with a tool call only when source_call_id matches. Keep multiple results in order. Unmatched step observations stay on the operation span; unassigned feedback remains structured in the reconstructed input without an invented role or tool association. Structured text and image parts remain serialized, but media bytes are not uploaded. ATIF v1.8 audio fields are unsupported.

Only LLM spans carry llm.* attributes. Trajectory final_metrics remain in agent-root metadata to avoid double-counting tokens. Producer-specific cache-write and reasoning token counts map to the corresponding OpenInference token-detail attributes when present.

ATIF timestamps are point events. Zero-duration LLM or TOOL spans can mean no unambiguous duration was available. Do not interpret them as proof that the operation took no time. Declared tool order does not prove serial execution.

ATIF discovery and conversion are best-effort. If the trajectory is missing or invalid, the plugin warns and records the run without a trace. A later replay cannot attach a trace to an immutable successful run.

Handle failures and resume

Selecting the plugin makes successful Phoenix recording required.

• Setup failures stop the job before trials run.
• Run or evaluation write failures stop the job. Already recorded trials remain in Phoenix.
• Harbor keeps terminal results that the plugin can ingest during resume.
• ATIF conversion or upload failure does not stop the job. The run is recorded without a trace.

Sequential resume and replay reuse matching datasets, experiments, successful runs, evaluations, and traces. Failed runs can be retried. If another job creates a newer version of the shared dataset, recover the original experiment and keep it pinned to its creation-time version. Do not run multiple ingesters for the same Harbor job because experiment recovery is not atomic across processes.

If Phoenix reports a conflict, do not tell the user to ignore it. The plugin validates the stored run's trial output and trace identity. A mismatch requires a new Harbor job or resolution of the conflicting Phoenix record.

Current boundaries

The plugin does not support:

• Harbor regrade jobs and other jobs derived from previous Harbor results;
• post-hoc ingestion of a completed job;
• live OTLP trace linkage, which is deferred to a follow-up;
• several configured datasets in one job;
• mixed configured datasets and direct tasks; or
• concurrent ingestion of one Harbor job.

For the public guide, use Phoenix's Harbor documentation. For Harbor command and task configuration, use the Harbor documentation.