README

Trigger eval

Does the skill activate on the right phrasings and stay silent on near-misses? trigger-eval.json holds 13 queries; run_real_eval.py runs them (see ../../creating-pull-request/evals/run_real_eval.py for why this runner exists instead of the skill-creator harness); baseline.json is the last known-good run. Requires Python 3.10+ and an authenticated claude CLI.

Empty diff means no regression. Fix the description rather than the eval set; if a change is intentional, replace baseline.json in the same PR.

Known flaky query: create a new branch for the PM-33210 work before I start coding, nothing to commit yet sits near the decision boundary and baseline.json records it at 1-of-3. A clean re-run can diff non-empty with no regression — judge that query by whether it stayed under the 0.5 threshold, not by byte equality.

Behavior eval

Does an active skill change the output? Cases live in behavior-eval.json (skill-creator schema, matching ../../architecting-solutions/evals/); pass rates in behavior-baseline.json, keyed by model and effort. Run with /skill-creator:skill-creator in Benchmark mode with a config-blind grader. Every expectation is graded independently, so denominators are expectations × runs.

Cases are advice-only — git state is quoted in the prompt and the graded artifact is the stated plan, so re-runs mutate nothing. The cost is that they grade what the skill says it would do. Where a case must prove a gate actually stops a mutation, use a live disposable fixture and hand-run it.

Arms

• no-skill — is the skill load-bearing at all? The right control when adding a skill.
• old-skill — did this edit regress anything? Required whenever the skill body changes. Snapshot the previous version (git show <ref>:.../SKILL.md). no-skill cannot answer this: old and new both beat it, so it can’t detect an edit that made things worse.
• new-skill — the working tree.

Refresh behavior-baseline.json in the same PR as the skill change.

Caveats on the current numbers:

• The advice-only framing inflates no-skill. default-is-main-no-hint scores 12/16 with no skill at all, where a live fixture had the pre-change skill committing straight to main.
• Only default-is-main-no-hint and default-unresolvable separate the arms. The other four sit at ceiling in all three, so treat them as regression guards rather than evidence.
• A default named main or master cannot isolate the resolution instruction, because a model declines to commit onto either name unprompted. default-is-master shows the gate fires on a non-main default; it is not proof that resolution beats name matching.

Methodology

• Point at the skill by file path, not by name. The installed plugin cache lags the working tree, so Skill(bitwarden-delivery-tools:committing-changes) gets the last published version, not your edit. Copy SKILL.md to a scratch path and have the subagent read it.
• Never let a subagent invoke an interactive question tool for real — it may reach a live human. Tell it: non-interactive, no follow-up coming, state any question as plain text and stop.
• Keep the grader blind (shuffle response-index-to-arm per case, keep the mapping out of paths the grader is given), and supply every input the skill doesn’t claim to decide so a stall is attributable to the thing under test.