Diagnostic Review

1. Record the case and knowledge boundary

Build a brief card: task/acceptance criterion → expected result → actual result → input/material state → versions/recent changes → available events/artifacts → impact. Identify provenance: user report, inspected code, trace, executed check. Do not call a retold trace your own observation.

A brief retelling or prompt fragment does not prove the full instruction, skill, or another component lacks a domain method. Do not make its incompleteness a separate Important finding without confirming the real execution path. It is a hypothesis to check; record a demonstrated output/validator defect independently.

Record known failure count and total examined cases; do not extrapolate available traces to others. Separately check that “successful” cases really meet the domain criterion. No trace means cause unknown; it does not yet prove system-wide absence of logging.

Build a short chain from externally observable data: input → selected method and actually loaded context → tool request/response → transformations → saved output → check → user response. Find the first observable discrepancy. Error origin may be earlier; downstream emptiness and the validator that missed it may be separate defects. Do not require hidden model chain of thought.

Minimize the example while preserving failure conditions: context length, event order, tenant/rights, versions, concurrency, retry/resume, external-service state. Failure disappearing after shortening is trigger evidence, not proof of poor task definition. Do not dismiss a single critical failure as noise; two repetitions do not prove cause. Plan probabilistic repeats proportionately to risk and available budget.

2. Select the layer and competing explanations

The table gives search directions, not ready-made diagnoses. A symptom can span layers; the model may also be a cause or interaction participant.

For a significant unclear cause, usually choose 2–4 plausible hypotheses, not every table row. Format: hypothesis → evidence for/against → remaining unknown → minimal discriminating check → outcome supporting or weakening it. A demonstrated defect needs no artificial alternatives. Choose the next read/test by explanatory discrimination, risk, and cost, not a familiar stack.

For context loss, check cold start or resume: from actually available and loaded materials, the agent reconstructs goal/user, work/validation method, current progress/next step, current decisions, and completion criterion. Adapt specifics to the domain: linter commands matter for a coding agent, not every agent. Compare source → addressability/access → loader selection → assembled context → observed answer. A file’s existence does not prove loading; a wrong answer does not prove the file absent. The fix may be in reading path, freshness, or model.

3. Check the check itself

Use validation-loop.md for completion/checker contracts; execution-continuity.md for consistent snapshots/handoff. Select additional discriminating cases by observed symptom:

For a fix review, reproduce the affected transition and a permitted control case. Correcting “done” reporting does not close the wrong-result cause; successfully stopping new tasks does not prove the fate of sent operations.

Storage, supervisor, budget, asynchronous-decision, and scheduling contracts are in sections 5–9 of execution-continuity.md. No separate service is not a finding when another component demonstrably performs the function. Diagnose mechanisms using actual configurations/traces; “environment matters more than model” does not exclude a model or interaction defect.

Trace requirement → checked result → source of truth → real validator/gate → negative-result handling → user status/response. Distinguish: no check; check not run; run on another path; unable to distinguish right/wrong; result ignored; stale artifact checked. A CI file, hook, or lint command proves neither execution nor substantive coverage.

For quality claims, apply evaluation-design.md: mandatory admission, valid alternative trajectories, measurement failures, coverage, comparability. A rising score with falling coverage does not prove agent improvement. For execution controls, check timing and covered paths under section 3 of architecture-contract.md: a check may run correctly but too late; a checkpoint may save state while permitting a repeated external action. Diagnosis names the violated obligation and mechanism, not just a missing component.

For a material criterion, provide a correct example and a plausible wrong one: wrong period, empty data, another tenant, nonexistent link, incomplete output with done=true. The check must distinguish them on independent grounds. A negative control must traverse the same gate/path under investigation; running a separate linter against an artificial error does not prove the production gate works. Intentional defects belong only in an authorized isolated copy/fixture; otherwise propose the test without executing it.

Model self-assessment and HTTP 200 do not certify outcomes. Programmatic postconditions check formalizable properties; a human may assess meaning/domain usefulness against an explicit rubric and source data. An LLM judge likewise needs validation on labeled good/bad cases and false-accept/reject measurement. A separate judge, E2E, or one overall score does not replace every evidence type. Check whether the agent can substitute criteria, test data, or final status to obtain green checks.

4. Design comparison and minimal correction

First separate cause localization from containing already demonstrated harm. Preventing false success is needed even while the reason for empty analytics remains unknown. It does not close data-loss cause. Correct the established mechanism; new memory, graph, judge agent, or model needs a testable benefit hypothesis and comparison with a simpler option.

For comparison, fix baseline version, tasks, state/data/tool responses, criteria, settings, and limits. Change one factor; when model and context assembler change together, compare all four combinations where possible to expose interaction. Stubbed tool responses localize downstream behavior but do not test correct request construction or live integration. Record that limit. Without the old baseline, do not confidently attribute regression to a component.

Disabling a component is a supporting hypothesis experiment. No change may mean redundancy, nonuse, or a weak sample; deterioration shows contribution under those conditions, not automatically the system’s main bottleneck. Do not disable real ACLs, approvals, cost limits, or other mandatory controls for diagnosis. Do not propose removing them based on a few ordinary successful tasks; an accepted contract and relevant-boundary checks are needed.

Compare domain correctness, false success/refusal, user effort, time, and completed-task cost. Retain previously successful regression cases and independent cases outside tuning. Do not impose a universal run count or mandatory model replacement as the final step: evidence, impact, and check cost determine order.

5. Change review and completion

For each significant correction, check: does it close the established causal link → on every affected path including bypass/resume → without weakening mandatory controls → what proves closure → which result requires rollback. Separate cause removal, consequence containment, and additional observability. Label owners and numerical thresholds proposed until agreed.

The report must enable action now: a confirmed defect/minimal correction or the next discriminating check with expected outcomes. Statuses: confirmed / refuted under checked conditions / hypothesis / check unavailable. For fix reviews distinguish designed / implemented according to code / behavior-verified; an experiment plan is not its result. Do not promise the whole regression resolved when only false success reporting was closed.

Stop when significant findings have evidence, a closure plan, or a precise gap with the minimum next check. Do not visit every layer for completeness or keep experimenting without a new discriminating question. Limit follow-up review to prior findings and changed paths; broader reconsideration requires new material evidence, not leftover checklist items.