Eval

Run them in this order: lint → render → diff → (if fail) strip.

Per-fixture flow

Reading diff/summary.json

Validated tier thresholds

Calibrated against actual Remotion + HF renders:

Each fixture’s expected.json carries:

• ssim_threshold — the gate for pass
• validation — the actual measured numbers from the calibration run
• translation_notes — what’s lossy and why

Critical: encoder config

Both Remotion and HF must output the same pixel format for SSIM to be meaningful. Remotion’s default JPEG output writes yuvj420p (full-range); HF outputs yuv420p (limited-range). The mismatch costs ~0.05 SSIM.

Every fixture’s remotion.config.ts sets:

If the user’s source doesn’t have these, add them in the translation step — otherwise the diff measures encoder differences, not translation fidelity.

What the noise floor looks like

The dominant non-translation noise is system font fallback divergence. Remotion’s bundled Chromium and HF’s chrome-headless-shell interpret font-weight: 800 differently when there’s no real font installed:

• Remotion HELLO at 160px: medium-weight stroke
• HF HELLO at 160px: heavy-weight stroke

This costs ~0.025 mean SSIM. Visible in T1’s frame strip. fonts.md covers how to mitigate (use Inter, load explicit Google Fonts).

Threshold rule of thumb

Set the threshold ~0.02 below measured p05:

• Real translation regressions drop mean by 0.05+ — caught.
• Encoder/font drift between CI runs is bounded at ~0.01 — not caught.

If a calibration run’s measured mean is far above your initial threshold guess, don’t tighten the threshold to fit. Leave headroom — fixtures re-rendered on different hardware will drift.

When the diff fails

1. Look at frame_strip.sh output first. A side-by-side strip at 6–10 evenly-spaced timestamps shows whether the failure is structural (wrong scene durations, missing element) or cosmetic (different font weight, slight timing skew).
2. Check diff/ssim.log. Per-frame SSIM tells you which frames failed. Cluster of bad frames in the middle of a scene = animation problem; bad frames at scene boundaries = sequencing problem.
3. Re-read the relevant reference. timing.md for spring/easing issues, sequencing.md for scene boundary issues, media.md for asset loading issues.

CI integration

The fixtures are not yet wired into CI (packages/producer/tests/ runs inside Docker; the skill corpus needs the same). PR 7 of the stack adds the orchestrator that runs all four tiers and emits an aggregated pass report. For now, evaluate by hand per fixture.