Commerce agent evals

The repo ships no eval harness; the suite is yours, because a case only means something against your catalog, orders, and fixtures. Paths below are in the reference repo: commerce_common/ is commerce-common/commerce_common/. Gate behavior that needs no model (provenance, caps, guardrails, approval) is unit-tested with FakeClient in commerce_common/testing.py; evals cover what the model decides.

The case shape

This block is the one home of the case shape and the scorer names; /author-commerce-evals refers to it.

{
  "id": "<flow>-<nnn>-<behavior>",
  "priority": "critical | high | medium | low",   "difficulty": "easy | medium | hard",   "tags": ["..."],
  "skip": "<reason, when a case cannot run yet>",
  "state": {"seen_products": ["..."], "cart": [...], "memory": [...], "staged_changes": [...]},
  "turns": ["<the customer's or operator's message>", "..."],
  "expected": {
    "calls_tool": ["..."],            "calls_one_of": ["..."],          "never_calls": ["..."],
    "first_tool": "...",              "first_tool_not": "...",
    "ui_components": ["..."],         "no_ui": true,
    "cart_contains": ["..."],         "cart_item_count": 0,             "cart_not_contains": ["..."],
    "staged_change_kinds": ["..."],   "no_applied_changes": true,
    "memory_contains": ["..."],       "memory_not_contains": ["..."],
    "skill_loaded": "...",            "skill_not_loaded": "...",        "no_skill_load": true,
    "reply_includes": ["..."],        "reply_omits": ["..."],           "max_tool_calls": 0,
    "rubric": "PASS if <condition>. FAIL if <condition>."
  },
  "notes": "<what the case pins and the fixture fact that decides it>"
}

state is the precondition; turns is one message unless the behavior under test is carrying state across turns; expected holds only the keys the case is about. Ids in state and expected are real ids from your fixtures. priority and difficulty let a report say which failures matter; a case that cannot run yet carries skip with its reason rather than being deleted.

Authoring rules

• Preconditions go in injected state (the products already seen, the cart, the memory facts, the staged queue), which the runner loads into the session state and the memory store before the turn. Earlier turns are for state the behavior itself carries, and for nothing else.
• Every positive has a negative: for each case that asserts a component, a skill load, a gate, a disclosure, or a memory write, a case in the same niche asserts its absence. A refusal case has a should-serve counterpart.
• Memory is three cases: a remark worth keeping is written; an identifier or excluded content is refused (nothing stored, no error to the person); a stored fact changes the next session's pick.
• Grade the final tool arguments and the state they produced: the ids on the last presentation call, the fields on the staged change, the cart and the memory store after the turn. The reply's wording is graded only for strings that must or must not appear. Which route the agent took (skill_loaded, one named component, never_calls on a presentation tool) is asserted only where the route is the behavior (a grounding read first, a write that must never happen); elsewhere calls_one_of names the acceptable set. When a live run takes a route the case did not expect and the answer was right, widen the case to the acceptable set; do not re-pin it to the route observed.
• A rubric is one PASS condition and one FAIL condition that no response satisfies both of; it names the fixture fact that decides it (the updated delivery date, the price today); variants you accept are written into it; it says nothing about tone, length, or the order components appear in.
• A turn that mentions health, a one-off errand, or hostile content asserts the memory end-state (memory_not_contains, or never_calls on save_memory). A case where a stored fact should change the pick uses a query whose results contain both the item the fact favors and the one it rules out; run the search before writing the case.
• max_tool_calls is set from what a well-behaved agent needs; a multi-item request fans out several searches in one round, and the present_suggestions call that ends the turn is not counted.

Scorers

• Code graders read the events a turn yields (commerce_common/streaming.py): tool_call names and arguments, tool_result with status blocked and the gate in reason, ui component names, the last cart_update or change_update, and the reply text. Every key above except rubric is a code grader.
• rubric goes to a judge. One judge call per dimension (budget respected, no invented availability, trade-off stated), returning structured output with a verdict and a reason; the transcript is passed to it as quoted material, tool results and component payloads included; when the transcript exceeds the judge's window, truncate from the start so the graded turn survives, and record the truncation on the outcome. Pin the judge model at temperature zero; a change to the judge model or a rubric invalidates every stored verdict scored with it, so the recording carries a fingerprint of both.
• A judge reply that does not parse into a verdict is a judge failure on the case, kept apart from an agent failure.

Run pattern

Diff failure sets; a topline moving a point between live runs is noise. A case that fails after a change means the change broke the behavior or the case encoded a stale one; fix whichever it is and say which in the commit.

Poisoned fixtures

Listings, reviews, and messages carrying instructions live in eval-only fixtures the runner merges into the backend for the run, under a third-party brand or seller; none of their ids appears in demo data, seeds, or captures. Each such case asserts the negative in code (never_calls, cart_not_contains, no_applied_changes, memory_not_contains, reply_omits), and every vector it asserts is one the driven turn actually puts in front of the model (a review is only read on a details call). Cover at least an instruction to write to the cart or stage a change, one to remember something, and a false claim (a code, a guarantee). The should-serve counterpart is a separate benign eval-only listing in the same niche, so an agent that refuses everything fails it.