Evaluation Procedure

Corpus source — the frozen manifest is authoritative

• The set of questions and BPs to assess is the frozen manifest from ACQUIRE_CORPUS. Do NOT rebuild it here, and do NOT use semantic search (aws___search_documentation) to enumerate questions or best practices. Semantic search returns ranked, non-canonical, sometimes stale results and is the source of the traversal-stall failures — enumeration happens once, in ACQUIRE_CORPUS, and is frozen.
• The only documentation retrieval in this step is reading an individual BP’s detail page at the bp_url already recorded in the manifest — and only when the BP’s title plus the workload evidence is not enough to judge status, or when writing a Critical/High recommendation. Prefer aws___read_documentation when the AWS MCP server is available; otherwise fetch that docs.aws.amazon.com URL over HTTPS with the environment’s web-fetch tool, and if no documentation access exists, judge from the BP title plus internal knowledge and disclose that limitation. Do not bulk-fetch BP detail pages.
• Canonical PILLAR##-BP## IDs are taken verbatim from the manifest. Never invent an ID; if the manifest lacks a BP you expected, that is a corpus problem to surface, not a gap to fill from memory.

Coverage strategy — assess every BP in the manifest

The purpose of a full review is comprehensive BP-level coverage: every BP in corpus/best-practices.jsonl receives exactly one status from the five-value vocabulary. The manifest counts (total, and per pillar) are your coverage targets — trust them over any number written in this skill.

Step 5a — The question inventory is the frozen manifest

Use corpus/questions.jsonl as the authoritative question inventory (its question_id, pillar, and title). This drives the per-question report table and the Step 5d audit. Do not re-walk the appendix or search to rebuild it — it is already validated and frozen.

Step 5b — Assess each pillar in its own scoped pass (MANDATORY for full coverage)

Why this pattern: when a single agent tries to assess every BP in one response, it can stop before finishing. Scoped to ONE pillar — with that pillar’s BP set already supplied by the frozen manifest — the agent assesses each BP without navigating the docs, and one pass per pillar aggregates to full coverage.

If your environment supports parallel subagents, dispatch one pass per pillar in the manifest in a single turn. If it does not, run the passes sequentially — one pillar at a time, completing that pillar’s table before the next.

Required output format (every pillar pass):

Row requirements:

• One row per BP in this pillar’s manifest set — the row count MUST equal the {N} header line (which is the count of corpus/best-practices.jsonl rows with this pillar_id)
• Status: exactly one of Implemented / Partially Implemented / Not Implemented / Not Applicable / Cannot Determine
• Risk Level: Critical / High / Medium / Low (or blank for Implemented/Not Applicable/Cannot Determine)
• Evidence: specific file:line references when code was analyzed, or “Based on description” when reviewing verbally
• BP ID in canonical PILLAR##-BP## format, copied verbatim from the manifest

Pillar pass instructions — use as the subagent prompt (adapt the dispatch syntax to your environment), or as your own working instructions per pillar when running sequentially:

Run it for every pillar in the manifest, using the pillar names and prefixes recorded there.

Cost/latency: per-pillar assessment uses more tokens than a single pass because each pass carries the workload context; with parallel dispatch, wall-clock is bounded by the slowest pillar. On slower-generating models a scoped pass also keeps each response small, which is where element-drop is avoided.

When to skip the per-pillar pass pattern: the user asked for a quick review / score mode / pillar-scoped review (those apply their mode adjustments), or a cost-constrained single-pass review is explicitly requested (be explicit it does not guarantee full-inventory coverage).

Step 5c — Aggregate the pillar-pass findings (PRESERVE citations verbatim)

Once every pillar pass is complete, merge their findings into a single structured report. CRITICAL: preserve every BP citation each pass produced. Aggregation is a merge, NOT a summary — do not paraphrase, cluster, or omit BP citations.

Aggregation rules — follow all:

1. Full BP Ledger required. The report MUST contain a “Full BP Ledger” section with a row per BP from every pillar pass, verbatim, no paraphrase.
2. No compression by pillar. Do NOT reduce a pillar pass to only its top findings. Every BP appears in the ledger. High-risk findings ALSO get a full-detail narrative in the “Critical and High Risk Findings” section — in ADDITION to the ledger, never instead of it.
3. Verify count before writing. Count distinct BP IDs in the Full BP Ledger section only (PILLAR##-BP## form) — do NOT also count the Critical/High narrative, which repeats ledger rows and would inflate the total into a false pass. It MUST equal the manifest total. If lower, you dropped some — add them back.
4. Cross-pillar patterns and prioritization are additive analyses that reference the ledger; they do NOT replace it.

Ledger row meaning by status: Implemented (workload demonstrates it, cite evidence) · Partially Implemented (gaps, cite the gap) · Not Implemented (absent, cite as missing — a valid, valuable finding) · Not Applicable (doesn’t apply, brief why) · Cannot Determine (evidence insufficient, state what runtime/interview data is needed).

Step 5d — MANDATORY coverage audit (do NOT skip)

Before producing the final report, self-audit and iterate if coverage is incomplete:

1. Count unique BP IDs evaluated (canonical PILLAR##-BP##), across all five statuses.
2. Compare against the target: the manifest total (corpus/best-practices.jsonl row count). Anything less is incomplete.
3. If below the manifest total, you MUST NOT proceed. Compare your evaluated set against the manifest, evaluate each missing BP (fetch its bp_url if the title is not enough), and repeat the count.
4. Continue until every BP in the manifest has an entry. A genuinely Not Applicable BP is marked Not Applicable with a one-line rationale — never silently skipped.

Audit output format (include before the executive summary):

If BPs evaluated is less than the manifest total, the review is not finished — return to step 3 above.

Retrieval economics per mode

• Quick review / score mode: assess at the question level using corpus/questions.jsonl; fetch a BP detail page only when a specific finding needs a canonical BP ID’s guidance.
• Pillar-scoped review: filter the frozen manifest to the requested pillar(s) and apply full BP-level detail for those pillars only.