Gate Templates

promotion-report.md Template

Every promotion run produces exactly one of these, committed alongside the checkpoint it evaluates. All four stages get a section regardless of where the run stopped — a stage the run never reached is marked NOT RUN, not omitted, so a REJECT report still documents the full gate.

Drift-Suite Scoring Table

Units: “pts” throughout this file and SKILL.md‘s Drift Budget table mean percentage points (absolute accuracy difference, e.g. 78% → 46% is 32pts), never relative percent change. State this explicitly in every report rather than leaving it implicit.

The frozen benchmark set and the drift budget it’s scored against — matches eval-harness-first‘s references/grader-templates.md drift-suite.yaml example exactly, so a report generated here diffs against the same frozen numbers that skill’s baseline file recorded:

Score every row in the frozen suite (general benchmarks plus the 200-500 domain-adjacent items) against this budget independently — a single domain-adjacent item breaching >5pt fails Stage 2 even if every general benchmark stayed within noise, and even if the checkpoint’s target-task metric improved substantially in the same run.

Sizing the Suite: The Honest Math, Then the Pragmatic Floor

The exact formula for a binomial accuracy metric’s 95% CI half-width: n ≈ 1.96² · p(1-p) / h², where h is the target half-width (as a fraction, e.g. 0.025 for 2.5pt) and p is the benchmark’s expected accuracy. SKILL.md‘s Drift Budget section calls for sizing n so the half-width sits comfortably under half the hard-fail threshold — for the 5pt budget here, h=2.5pt (0.025). Worked at a typical GSM8K-style accuracy of p≈0.7:

The strict n for a 2.5pt half-width at p≈0.7 is ~1,300 — not 200. A 200-item suite at that same p only achieves:

n=200 is a pragmatic floor, not a suite that meets the strict 2.5pt target. Use it when a ~1,300-item domain-adjacent pool doesn’t exist for the task (the common case), but with three rules attached:

• Report the actual CI half-width alongside every Stage 2 verdict — at n=200, p≈0.7 that’s ≈±6pt; recompute per benchmark since p varies row to row.
• A hard-fail margin smaller than the reported CI half-width makes the verdict REJECT (uncertain), never PASS or HARD FAIL. A delta that clears or breaches the 5pt budget by less than the half-width has not actually been distinguished from noise at 200 items. Resolve REJECT (uncertain) with either a larger suite (build the domain-adjacent pool toward the ~1,300-item strict number) or a same-config seed-repeat (worked example below) before promoting — do not round an uncertain result to whichever verdict is more convenient.
• 200 is a floor to derive from the budget, never a fixed constant to copy. A tighter hard-fail threshold than 5pt needs the formula re-run at the new h, and n=50 is well below even the pragmatic floor: at p≈0.7-0.8, n=50 carries a half-width around ±13pt, wider than the entire hard-fail band.

Cautionary Worked Example: A Real 5-Run Trajectory That Was Mostly Noise

From a dogfood run gating LoRA checkpoints on a frozen n=50 GSM8K drift slice, then re-measuring the same checkpoints at n=200 once the pattern looked suspicious:

Read at n=50, this trajectory looks like real signal: replay helps (+18pt), LR helps further (+8pt), more replay hurts (-26pt) — a story worth writing a remediation note about. Read at n=200 for the two points actually re-measured (r2 and r5, same config, different seed): 60.0 vs 58.5, overlapping Wilson CIs — statistically indistinguishable, against a base-model score this run family never got within ~23pt of at either n. The entire 46→64→72→46 shape at n=50 was sampling noise riding on top of one consistently large true drift. Two lessons this motivates in SKILL.md‘s escalation-ladder caveats: (1) a per-checkpoint verdict at n=50 is still a fact about that checkpoint on those 50 items, but (2) the run-to-run remediation trajectory built from a sequence of n=50 verdicts is not a reliable guide to which lever worked — re-run the suite at budget-derived n before trusting a multi-run remediation story, and treat single-seed lever attribution as a hypothesis until a same-config seed pair confirms it.

Paired-Arena Protocol

• N items: 200 minimum, drawn from the same domain-adjacent pool used in Stage 2 (not the training set, not the goldens used to build the checkpoint). This is the same pragmatic floor as Stage 2’s sizing rule above, not a suite proven to resolve the margin below at 95% CI: near the 50% boundary, n=200’s half-width is 1.96 × sqrt(0.25/200) ≈ 6.9pt, wider than the 5pt margin threshold. Report the CI half-width alongside the win rate; if the margin is smaller than the reported half-width, the stage verdict is REJECT (uncertain) — same rule as Stage 2 — resolved with a larger arena or a same-config seed-repeat before promoting. This 200-item minimum is for the LLM-judge protocol. The deterministic variant below has its own, smaller floor because it isn’t subject to judge noise on top of sampling noise.

Deterministic Variant (No LLM-Judge)

When every grader in the harness is deterministic (code-based, no judge anywhere), Stage 3 collapses to a per-golden paired comparison instead of a judge-scored arena:

• Pairing: for each item in eval/goldens.jsonl, compare the checkpoint’s graded verdict against the baseline’s verdict on the identical prompt (from runs/baseline/results.json, paired by task_id). Win = checkpoint passed where baseline failed; loss = the reverse; tie = same verdict either way.
• N items: every golden, not a 200-item minimum — the goldens set itself is the population, not a sample drawn from a larger pool. If the goldens set is small (dogfood scale, <100 items), say so in the report and treat the resulting CI width as evidence quality, not grounds to pad the count with unrelated items.
• Position randomization: N/A — deterministic grading is order-independent, so there’s no position bias to correct for.
• Judge calibration: N/A — no judge in this path.
• Tie handling and win-rate threshold: same as the judge protocol below (ties count half a win each; 50% + margin, CI excludes 50%).
• Position randomization: for each item, flip a coin on which of {CHECKPOINT, BASE_MODEL} appears first in the judge’s prompt; log the raw assignment. A judge with any positional bias otherwise inflates whichever model is shown first, independent of quality.
• Judge: pinned snapshot, calibrated per eval-harness-first‘s references/judge-calibration.md — an uncalibrated judge here invalidates the whole stage.
• Win-rate threshold: checkpoint must win 50% + margin (5 points is a reasonable starting margin) with the 95% CI excluding 50% — a checkpoint sitting at 51% with a CI spanning 44-58% has not demonstrated a win, it has demonstrated a tie.
• Tie handling: judge ties count as half a win for each side in the win-rate calculation, not as excluded items — excluding ties inflates the apparent win rate.

Replay-Mix Configuration Example

The standard catastrophic-forgetting mitigation — general-domain data blended into the target-task training set at 10-30%. The escalation order (when and how far to move this fraction) is owned by SKILL.md‘s Catastrophic Forgetting section — this file gives the config shape and the swap-not-add mechanic, not a competing order.

Start at 20% when no prior forgetting data exists for the task. When a later run needs to move this fraction, swap rows rather than adding them — drop target-task rows out of the mix as replay rows go in, so total row/step count holds constant between runs. Adding replay rows on top of the existing set changes replay fraction and total optimizer steps in the same move, which makes it impossible to attribute a drift-score change to either variable alone — see dataset-curation‘s replay-mix construction recipe (references/synthetic-data.md) for the full swap procedure. Do not assume the dose-response is monotonic: a documented dogfood run saw 30.4% replay (added, not swapped) score 18 points worse on the replayed capability than 20% replay at otherwise-identical config, because the added rows also raised total steps 50→58. Only drop toward 10% once a run at 20% clears Stage 2 with headroom (all deltas comfortably inside the noise band, not just under the hard-fail line).