Runtime Harness

If any step below produces a PHP fatal, STOP. Later steps won’t produce meaningful output. Capture the failure, escalate to the plugin’s implementer to fix, then re-run from step 1.

A WP_Error return is acceptable — any <plugin>_* or upstream-prefixed error means the execute callback handled the error path gracefully. Only PHP fatals block.

Step 0 — identify the env-up command

Read the plugin’s AGENTS.md for the canonical env bring-up command. Do NOT assume npm run wp-env start works for every plugin. Common patterns seen in real plugin trees:

• npm run wp-env start — projects using @wordpress/env with a checked-in .wp-env.json.
• npx wp-env start — projects using @wordpress/env without a custom npm script wrapper.
• composer install && composer test-php --setup-only — package-local test bootstraps that don’t run a full WordPress install.
• docker-compose up -d — plugin-specific dev Docker stacks.

Plugin families with their own dev tooling will have their own bring-up command in AGENTS.md; follow it as documented.

If AGENTS.md doesn’t document it, ask the user rather than guessing. Record the env-up command + the corresponding wp-cli invocation (e.g. npx wp-env run cli wp) and use them uniformly for the rest of the harness.

In this file, <env-cli> is shorthand for whatever wp-cli invocation the plugin uses.

Step 1 — bring up the env and sanity-check

Confirm:

• WordPress version >= 6.9 (Abilities API available in core).
• The plugin being verified is active.
• wp_get_abilities exists (true if WP >= 6.9, else the Abilities API feature plugin/package must be active).

Any “no” answer halts the harness.

Check 1 — ability names match source-expected list

Enumerate runtime abilities and diff against the static inventory from static-enumeration.md:

Compare against the static inventory:

• Source contains ability, runtime missing → FAIL. Registration hook isn’t firing; check init hook timing and plugin activation.
• Runtime contains ability, source missing → WARN. Dynamic registration path the enumerator couldn’t follow. Document but don’t block.
• Counts match → OK.

Check 2 — annotations read back as declared

Cross-reference each annotation against the audit’s declared value (if an audit was provided) AND against the static inventory’s declared value. Mismatch on either axis → FAIL.

This check complements the static adversarial check from annotation-correctness.md: static checks the callback’s actual behavior; runtime checks what the registration hook resolved to at boot time. Both must agree for the annotations to be trustworthy.

Check 3 — each read ability’s execute() behaves as the contract claims

Two verification levels. The smoke level catches bootstrap and gross-error regressions; the high-confidence level is the one that actually exercises the ability against the data shape it will see in production. Run both when the audit doc provides seed_data_needs; run the smoke level alone when it doesn’t.

Level 1 — smoke execution (synthetic inputs)

Confirm each read returns OK or a vocabulary WP_Error. Catches PHP fatals, un-bootstrapped services, registration failures.

Acceptable outcomes:

• OK — the ability returned without error.
• WP_Error(<plugin>_not_initialized) — bootstrap guard fired (e.g. un-bootstrapped service). Happy error path.
• WP_Error(<plugin>_<resource>_data_unavailable) — transient backend error. Acceptable.
• WP_Error(<upstream_code>) — upstream third-party error bubbled through. Document, don’t block.

Unacceptable:

• PHP fatal → stop the harness.
• WP_Error with a non-vocabulary code → WARN. Cross-reference ../../wp-abilities-api/references/error-code-vocabulary.md.

Abilities with required input get invoked separately with a synthetic-but-plausible value:

A synthetic ID on a fresh install typically triggers <plugin>_<resource>_data_unavailable or an upstream-equivalent code. Both are acceptable for Level 1 — they mean “ability dispatched cleanly, the backing reported no data,” which is the smoke signal.

Level 2 — high-confidence verification (representative seeded data)

Synthetic inputs catch fatals and gross errors. They do NOT catch wrong IDs returning the wrong record, cached sentinel values being served instead of fresh data, permission gates appearing correct against a synthetic ID but not against a real one, filtered labels diverging from the unfiltered claim, missing capabilities surfacing only when a real record is in scope, or curated output drift (a new field added to the controller that the ability inherits and now leaks). These are the failure modes that matter and they only fall out when the ability runs against the data shape it will see in production.

For each ability whose audit entry declares a non-null seed_data_needs, seed the environment per that field, then call the ability with representative inputs (a real id, a real slug — not a synthetic placeholder), and assert the output shape AND the privacy contract:

Adapt $expected_keys and $forbidden_keys to each ability. The audit doc’s return_type field hints at the shape; the privacy keys depend on the plugin’s domain. For payments-family plugins, full PANs / bank numbers / raw tokens are the canonical forbidden set; other families substitute appropriately. The plugin’s in-tree contract tests (the overlay’s test-the-public-contract.md for WooCommerce extensions) are the durable home for these assertions on every CI run — this Level 2 check is the one-off harness run that produces the PR artifact.

When the audit doc declares seed_data_needs: null, Level 2 is not yet runnable: the auditor has not identified the seed shape. The harness reports LEVEL 2: pending (seed_data_needs is null — ask the implementer) and proceeds. When seed_data_needs is a string, the harness operator seeds per that description before running the representative-input block above.

Check 4 — each write ability’s missing-input returns ability_invalid_input or <plugin>_missing_<field>

Acceptable codes, per ../../wp-abilities-api/references/error-code-vocabulary.md:

• ability_invalid_input — the Abilities API’s schema validator fired first (normal REST-bridge path; emitted by WP_Ability::validate_input() in core).
• <plugin>_missing_<field> — the execute callback’s own guard fired (direct-invocation path).
• <plugin>_invalid_<field> — same, for the wrong-type case.

UNEXPECTED_OK → FAIL. Validation is missing; the ability accepted no-input and proceeded to do something it shouldn’t have.

Check 5 — permission gate denies subscriber, allows admin

Expected:

Any inversion is a bug in the registered permission_callback. See permission-roundtrip.md for the deeper cross-checks (audit’s declared capability → registered callback → resolved current_user_can(...)).

Public abilities (deliberately ungated) expect subscriber: true AND admin: true. Verify that the ability’s permission_callback is '__return_true' in source before accepting this outcome.

check_permissions() returns bool|WP_Error per WP_Ability::check_permissions() in WordPress core. A WP_Error with code ability_invalid_permission_callback means the registration didn’t supply a valid callable — a hard FAIL. A WP_Error with code ability_callback_exception means the callback threw — also a hard FAIL.

Check 6 — twin-invocation heuristic for idempotent abilities

Only apply this to abilities annotated idempotent: true whose execute() returned without error in Check 3. Per annotation-correctness.md step “Runtime check complement”, this is a heuristic: idempotent in core means “no additional effect on the environment” (per the idempotent annotation’s docblock in class-wp-ability.php), not “byte-identical return values.”

Interpretation:

• match=true → cheap PASS. Same input produced the same response, and any environmental writes (write abilities) were the same on both calls.
• match=false → inspect what changed before deciding:
  • Response embeds a per-call timestamp, nonce, or random ID → environment unchanged. Still idempotent under core’s reading. Optionally remove the field if the agent doesn’t need it; the annotation stays idempotent: true.
  • Response reflects a counter or sequence that grew between calls → real environmental change. FAIL: drop the idempotent: true annotation or fix the underlying write to be input-determined.

For ambiguous cases (response varies but no obvious counter), supplement with a state diff: snapshot a representative table or option before call 1, snapshot after call 2, diff. If state changed by more than the input writes would explain, the ability is non-idempotent.

Output format

The runtime harness writes a dedicated section in the run report:

When the harness catches a bug

Observed pattern:

1. Harness surfaces a PHP fatal (e.g. ArgumentCountError: Too few arguments to function <controller>::__construct).
2. Implementer fixes the bug in a focused commit.
3. Harness re-runs; write the post-fix output as the headline status.
4. Preserve the pre-fix trace in the report under a “Pre-fix status” section so reviewers can see what verify caught.

This is the highest-leverage signal the runtime harness produces: bootstrap-ordering and missing-dependency bugs that pass static review because the source declares the right shape — they only manifest when the registration runs against a booted WordPress.