EVAL

Use AXIS as the skill-eval harness.

Do not add custom run-codex.* scripts. Add a small AXIS scenario, fixture, or artifact pattern instead.

Quick Run

Run the working set:

Run one case:

Inspect results:

Compare to a stored baseline:

Set or refresh a baseline only after reviewing the report:

Evaluation Principles

Use these principles when adding or changing skill-writer evals:

1. Start with 2-3 realistic cases before expanding the suite.
2. Compare candidate output against a baseline instead of trusting one-off scores.
3. Write observable judge checks: files created, forbidden files absent, validation mentioned, generated instructions concise.
4. Use LLM-as-judge for open-ended quality: concision, routed structure, source coverage, progressive disclosure, and overfitting risk.
5. Require concrete evidence in judge checks and manual review: paths, artifacts, diffs, transcript snippets, or validation output.
6. Track timing, token use, and interaction waste from AXIS reports.
7. Review transcripts when a run fails or is slow; wasted steps usually point to bloated or ambiguous skill instructions.
8. Use blind old-vs-new comparison for qualitative regressions when scores are close.

Case Set

AXIS scenarios live in evals/scenarios/.

Add provider-specific cases only when changed rules affect provider-specific mechanics.

What To Test

AXIS Setup

evals/axis.config.json configures:

1. agents: [{"agent": "codex"}] so Codex is the agent under test.
2. skills: ["./.."] so AXIS installs this local skill-writer skill for the run.
3. scenario limits so a stuck generation fails quickly enough to debug.

Each scenario configures artifacts for generated skill files, registration files, and validator output when present.

Each scenario copies the repository into AXIS’s isolated workspace before Codex starts. That gives skill-writer the same repo conventions it sees in normal use without polluting this checkout.

Rubric

Score each dimension pass, warn, or fail during manual review. AXIS supplies the transcript, artifacts, and judge scores; this rubric decides whether to adopt the change.

Critical dimensions: trigger precision, artifact minimality, runtime concision, progressive disclosure, and validation.

Adoption Gate

Adopt a skill-writer change only when:

1. AXIS runs complete for the working set.
2. no critical dimension regresses on holdout cases.
3. at least one target dimension improves on the case that motivated the change.
4. structural validation passes for generated skills or the transcript explains a legitimate blocker.
5. runtime SKILL.md remains a router and does not link to eval files.
6. source and maintenance notes are updated only when the skill contract changes.

Result Template