Setting the file. One moment.
Subchapter 8.13
references/validation-loop.mdMarkdown7 KBView on GitHub
Read when designing acceptance, release, and feedback, or diagnosing unconfirmed outcomes. These are functions of one architecture, not mandatory separate services. A simple task needs a compact description of applicable checks; explain non-applicability rather than creating empty chapters. The check set does not enlarge the interview or replace domain methods.
For each applicable function, populate input/version → property checked → mechanism and owner → execution timing → pass/fail/unknown → consequence → evidence. Show what the platform supplies, what is proposed, and what is unknown. “Not applicable” differs from a missing mandatory check.
| Function | Architectural decision | Diagnostic check |
|---|---|---|
| Eval Registry | Versions of scenarios, oracle/rubric/scorer, run conditions, thresholds/admission criteria. Capabilities, regressions, escalations; long horizon when work is long-running | Which scenarios were actually executed/evaluated versus excluded; whether versions/denominators are comparable |
| Review | Who checks domain meaning, architecture, or artifact quality: program, model, human, or combination. Task-specific checklist and remediation bound | Whether the checker distinguishes good from plausible bad results and has enough primary data |
| Security Gate | Applicable asset-based threats/checks. For code: SAST/DAST, secrets/dependencies by risk; for text: data protection and output admissibility | Whether real paths are covered and dangerous effects/disclosure can precede checking |
| Policy Gate | Check concrete action, principal/tenant, resource, parameters, permissions, extra conditions; enforcement point | Parameter substitution, approval expiry/revocation, alternative handler, retry/resume; a negative verdict actually prevents action |
| Evidence Bundle Gate | Completeness, relevance, and freshness of evidence for the claimed outcome | All fields present but tests refer to an old version; effect unknown; links inaccessible or do not support claims |
| Release Gate | Admit a composite version to active use: owner/policy, checks, bounded pilot, stop/rollback | Whether an unverified version can activate; configuration/data/continuing-task state consistency |
| Telemetry Feedback | Operational signal → diagnostic case → evaluation/requirement → targeted change → comparison → accept/rollback | Whether failure becomes reproducible, results improve outside tuning, and criteria were not changed for a green score |
Order depends on effects: write admission is checked before writing; result confirmation afterward. The diagram does not prescribe sequential execution of all seven components at every step. The registry may be a versioned test directory; review an existing check or a human; release for a local assistant a controlled version replacement. Platform documentation does not prove configuration or coverage.
Without a mandatory verdict, the result is not admitted. Choose a concrete outcome: waiting, stopping, or permitted degradation. Evaluator failure does not prove a domain-result error. Use evaluation-design.md and diagnostic-review.md for axes, measurement failures, and positive/negative controls.
Design one verification package or manifest linking authoritative records. Choose format by environment; a separate service is unnecessary. For a material criterion, link:
task/run ID → R/E-ID → composite agent and material input versions → output ID/version → check/scorer and verdict → evidence link → completion status/unknown.
For actions, add operation identity, exact parameters/payload version, applicable authorization/policy decision, and confirmed effect state. For operations, add costs with accounting coverage and a link to subsequent user-value measurement. Composite version includes model, prompts/skills, tools, policies, runtime, and material data. Label open values explicitly.
Check:
The model cannot grant itself final admission by writing done=true. For a proposed agent, the bundle specifies future evidence and a populated synthetic example, not real-run results. The final PDF under result-delivery.md presents findings and evidence links but does not replace evidence.
Synthetic run r18: requirement E4—a correct report for period P; input snapshot S3, method v2, output report R4. Numeric checking compares R4 with S3; domain review checks causal-inference limits. MAU changed from 1000 to 1100; no randomization, and prices also changed. The +10% calculation may pass while the experiment’s contribution remains unconfirmed. The next-experiment decision rests on limitations, not an invented causal effect.
If R4 is replaced with R5, successful R4 checks do not admit R5. If R5 publication times out, the final status is “report prepared; publication unconfirmed,” with operation ID and the next authorized reconciliation step. Separately record the reason if publication was never authorized.
If a Guardian/Review-Agent is planned, distinguish observation / pre-effect interception / correction. For each mode, define visible data, intervention point, authority, timeout, unavailable behavior, cost, and its own false-accept/false-block checks. Correction is a separate action subject to normal admission rules; reviewer authority does not automatically expand.
Example: a write completes at t15; Guardian returns deny at t20. The monitor detected the event, but pre-write prohibition was not enforced. To verify a correction, send prohibited and permitted operations through the actual dispatcher/adapter: the former never reaches an external effect; the latter reaches it and is confirmed. Test Guardian unavailability separately. A model judge does not replace mandatory authorization or guarantee attack absence.
Review is bounded by agreed rounds, time/cost, and stopping rules. Unresolved disagreement becomes a specific open decision or escalation, not endless exchanges between worker and reviewer. Do not disable mandatory controls for speed or a diagnostic experiment.