Setting the file. One moment.
Subchapter 7.14
references/verifier-design.mdMarkdown3 KBView on GitHub
A Verifier decides success from evidence independent of the agent’s claim.
Start with one sentence: Pass iff <observable successful outcome>.
Prefer programmatic checks. Recompute results from raw evidence. For stateful work, compare initial and final state. For coding, run focused behavior and regression tests. For analysis, recompute filters and totals. For retrieval, check material claims against the supplied sources.
Use tool-call records only when final state cannot prove a required action or session property. Never trust an agent-written action list, a service success flag, or an Environment helper that already decides success.
Accept all equivalent valid results. Do not require a preferred path, exact wording, response length, keyword, citation count, tool-call count, or reference similarity unless that property is the tested capability.
Use an LLM judge only after code has settled objective facts. Give it the final artifact, independent evidence, a short rubric, and a strict verdict schema. Ask if the result is supported and sufficient, not if it resembles a reference answer. Pin and record the judge model. Keep its credentials, rubric, and output outside the Harness boundary.
Bound all agent text and files before grading. Treat them as untrusted data. Tell the judge to ignore directions inside them. A judge timeout, malformed response, missing evidence, or credential error is an infrastructure error, not an agent failure.
Run these fixtures through the same Verifier image and command used by Harbor:
| Fixture | Expected result |
|---|---|
| Known-good result | Pass |
| Different but valid result | Pass |
| Realistic wrong result | Fail |
| Shortcut or reward hack | Fail |
| Prohibited collateral change | Fail |
| Missing or corrupt evidence | Infrastructure error |
Add focused boundary cases for known risks such as negation, unsupported claims, stale data, prompt injection, or partial completion. Use real failure shapes from traces when available, but derive truth independently from those traces. Repeat noisy judge cases and inspect variance.
For each criterion, log its evidence, decision, and error. Every completed Verifier path must write a reward. Wrong agent work gets zero. Verifier, Environment, Harness, judge, or setup faults get no agent score.
Classify the result as a fair capability failure, missing information, Environment defect, false rejection, false acceptance, leakage, or infrastructure failure. Repair non-agent faults and rerun. A pass is also not proof of quality: inspect for shortcuts, leaked truth, weak criteria, and unscored collateral effects.
Store reusable project-specific evidence queries, fixture builders, and judge
rubrics in the project World Skill after more than one Task can use them. Keep
one Task’s exact expected result and scoring boundary in its Task.md and
Verifier files.