Calibration

Do not require a weak-target-strong ladder or a fixed model count. One carefully inspected target-model run can be sufficient when deterministic checks pass and the run answers the current quality questions.

Do not require a fixed pass-rate pattern. A weak model can pass by finding a leak. A strong model can fail because of a broken service or unfair prompt.

Read complete runs

For each selected trial, inspect:

• all messages, model calls, tool calls, results, retries, and errors;
• initial and final Environment state and external effects;
• setup, service, readiness, reset, timeout, and cleanup logs;
• every Verifier criterion, its evidence, decision, and error; and
• resolved Harness, model, Environment, judge, and timeout settings.

Compare strategies, not only rewards. Check whether the agent had all required information, whether the Environment behaved as designed, whether the Verifier accepted equivalents, and whether a pass used a shortcut or leaked fact.

Classify each problem

Do not make a Task harder to hide a defect. First repair all non-agent causes and rerun affected trials.

Judge difficulty and fairness

A good hard Task requires real work. It does not rely on hidden information, unclear language, fragile startup, unrealistic data, or a broken check. If all models fail, test solvability and fairness first. If all pass, inspect for weak criteria and leakage before changing difficulty.

To change difficulty, change one supported condition: more relevant state, longer history, stale or conflicting facts, permissions, delayed effects, required clarification, or collateral-change risk. Keep the required evidence visible or normally discoverable.

Report model and Harness versions, trial counts, criterion results, strategies, failure causes, defects fixed, rerun results, remaining uncertainty, and the next recommended change.