Methodology

The VM boot is the unit of replication

JIT state, code layout, GC rhythm, host hardware, and phase order are random effects fixed at boot time. Iterations within a boot share all of them, so they are not independent samples — treating them as independent produces confident nonsense (a p<1e-4 effect in one run that vanishes in the next). The harness therefore:

• computes each metric’s delta per boot (mean of that boot’s paired within-boot deltas), and
• runs t-based inference across boots (bench-stats.mjs); the CI and p-value you see are boot-level.

The within-run p printed in brackets is a diagnostic only. It tells you the pairing worked; it cannot support a claim.

Allocation follows from this: more boots × fewer runs beats fewer boots × more runs at fixed budget, and boots run in parallel — adding them costs money, not wall-clock, so don’t be shy with them when sensitivity matters (CI shrinks with sqrt of boots). Defaults: e2e 16 VMs × 1 block × 2 runs. Both arms always run in the SAME VM, interleaved ABBA, paired per (vm, run) — hosts differ by up to ~20%, so cross-VM comparisons are meaningless; VMs exist to replicate, not to compare.

Claims

• A claim requires boot-level p < 0.01, reported with its CI, on a team/config whose boot-level inference an A/A run has validated.
• Anything that will drive a decision (merge, revert, ship) gets one independent confirmation cell first (standard perf-CI practice — Mozilla/Chrome retrigger, MongoDB change-point detection — nobody alerts off one detection).
• Everything else is “noise-compatible”, reported as no detected difference — not as a small effect.

Percentiles

A per-run p95/p99 from a small load phase is a near-max order statistic, not a distribution estimate. Compare percentiles only at boot level; p99 needs ~1k+ samples per boot to mean anything at all. Tail effects are also where phase carryover lives (a route’s serial tail depends on the previous route’s load phase) — investigate tails with --isolate-routes, which restarts the server between routes.

A/A validation

Before claiming anything on a new team/config/suite, run an A/A cell (same ref as both arms) at default allocation:

The A/A validates the inference itself. With identical arms, expect about (number of metric-cells x 0.01) cells at p < 0.01 by chance — under 1 for the e2e suite’s ~24 cells, roughly 1 for the ssr suite’s ~88. The pass criterion is therefore two runs: a suite fails its A/A when any cell reaches p < 0.01 with the same sign in both (a bias tied to the arm label recurs; chance does not), or when any single cell lands beyond the familywise bar (p < 0.01 / number of independent variant-x-phase cells). A failing cell means boots are sharing label-correlated conditions and the CIs are too narrow — find the mechanism before claiming; a magnitude gate is not a substitute. The A/A CIs also tell you what effect sizes this config can resolve. Re-run the A/A when the platform changes (VM hardware generation, node version, bench app or fixture changes).

Validated so far: e2e (twice, 0/24 cells), ssr (two-run criterion: one chance cell at p=0.0024 in run 1 dissolved to p=0.86 in run 2, nothing recurred; runs ssr-aa2/ssr-aa3 in the results db).

Interim progress display

The launcher’s progress digest shows, per route/phase, the running rps effect and a directional confidence — P(candidate is actually faster/slower | boots so far), the Student-t posterior under a flat prior (not a p-value). It appears from 4 completed boots. Display only: runs complete their planned allocation, and claims come only from the final fixed-n analysis.

Fingerprints

Every result row records a build fingerprint (dist-file hash + version string) read from the tree that actually served requests. The analysis header verifies them: inconsistent fingerprints within an arm invalidate the run (a VM measured the wrong build). Identical fingerprints across arms with different version strings can be legitimate — the hash covers the compiled server files, so client-only changes don’t move it; check the version strings before declaring an accidental A/A.

Honest reporting

• State the platform (e.g. “x86 Xeon sandbox VMs”) next to any number. Magnitudes are platform-dependent — GC’s share of request time scales with CPU speed — so direction and mechanism transfer between platforms, percentages do not.
• Print per-boot values; if boots disagree in sign, the mean is not a finding, and say so.
• Never conclude anything from the live streaming estimates. Peeking at sequential results and stopping when p looks good is a classic way to manufacture false positives.