Setting the file. One moment.
Skill 32 · Managing Workflow Secrets
Subchapter 32.11
evals/README.mdMarkdown3 KBView on GitHub
This directory holds the eval set for the managing-workflow-secrets skill: the durable,
re-runnable case definitions plus a single dated snapshot of the last validation run.
evals/
README.md # this file
evals.json # THE eval set — case definitions (id, name, prompt, expected_output, files)
files/ # input workflow fixtures each case starts from
results/
SNAPSHOT.md # dated, point-in-time validation report (benchmark + ablation + triggering)evals.json + files/ are the load-bearing part: the specification you re-run whenever the skill
changes. results/SNAPSHOT.md is a historical report, not a live result — see the warning at the
top of that file.
| ID | Name | What it exercises | Fixture |
|---|---|---|---|
| 0 | add-secret-retrieval | Full AKV+OIDC lifecycle wired into a job; consume via step-scoped env: | build-and-push.yml |
| 1 | downstream-job-handoff | Secret must not cross a job boundary; re-retrieve per job | deploy-cant-see-secret.yml |
| 2 | flag-secret-exposure | Flag echo / file-write / CLI-arg exposure of a retrieved secret | leaky-secret.yml |
| 3 | reusable-workflow-triad | Cross-repo reusable workflow: forward the OIDC triad explicitly | caller-workflow.yml |
| 4 | multi-secret-folded-list | Three-plus secrets rendered as a folded block scalar, one per line | multi-secret-deploy.yml |
| 5 | app-token-cross-job | Mint a short-lived GitHub App token instead of crossing a raw secret | app-token-cross-job.yml |
| 6 | logout-live-session | azure-logout placement exception when a later step needs the session | build-acr.yml |
| 7 | ask-for-names | Never infer vault/secret names — use placeholders and ask | deploy-datadog.yml |
There is no runner committed here. Each case is: feed the model the prompt plus the named
files/ fixture, with and without the skill, then grade the output against expected_output.
Because nothing in the repo regenerates results, results/SNAPSHOT.md can only go stale — re-run
before trusting it, and expect it to lag evals.json (evals 4 and 5 were added after the snapshot
and have never been run).