Setting the file. One moment.
Subchapter 14.9
references/failure-modes.mdMarkdown11 KBView on GitHub
Things that broke during skill development. Check against this list before you ship.
Also bundled
GitignoreRunning the matting ONNX through the CoreML EP partitions the graph across providers; the mixed-precision boundary produced alpha ≈ 30 (out of 255) inside the subject’s face while the background correctly read 0 (observed with the previous RVM engine — don’t re-try). Result: caption text visibly shines THROUGH the face.
Fix: Only ever use providers=["CPUExecutionProvider"] for the matting ONNX. Our script does this; don’t “optimize” it by adding CoreML.
(historical — the PP-MattingV2 engine was replaced 2026-06-12 by hyperframes remove-background; kept because the lessons generalize) Two earned scars from the ONNX era: (1) fixed-size model exports — squashing a portrait clip into a landscape canvas distorts humans; contain-pad (aspect preserved, centered, alpha cropped back) instead. (2) sharp returns a 3-channel buffer from .raw() after resizing a 1-channel raw input — reading it with a 1-channel stride silently produces a smeared, striped, displaced matte that can look plausible. Always toBuffer({resolveWithObject:true}) and stride by info.channels (other scripts in this skill still use sharp).
Fix: both handled inside matte.cjs; preserve them if you touch it.
rembg’s general/human models capture the person but drop handheld props (mic, notebook). If the caption crosses the mic, it will NOT be occluded by the mic — text floats in front of it unnaturally.
Fix: For scenes where props matter, either (a) accept that the human-matting model captures what it captures, or (b) add a manual mask ROI for the prop region. We haven’t productized (b) yet.
First champion iteration put all cap elements inside a flex column with display: block. Hidden captions (visibility: hidden) STILL reserve flex space. Result: cg-4 “dreaming” got pushed to y=700, landing right on the hand gesture zone and getting clipped.
Fix: Inside the caption plane, each .cap is position: absolute at the same coords. Hidden ones don’t occupy layout. Both templates in this skill do it this way.
First champion iteration used a full-width header band across the TOP of the frame (top: 30, left: 40, right: 40, height: 360, rotateX: -3deg, justify-content: flex-start, align-items: center). Subject’s head crosses the center, eating the middle of every caption. “know, for me” became a faint “now, for” with the letters sliced apart.
Fix: On close-crop podcast shots, don’t use symmetric full-width bands. Use corner-column-crown with caption column in the corner opposite the head.
If you set align-items: flex-end on the plane and don’t constrain width, each .cap shrinks to content width. Long words then extend left beyond the plane’s “visible” box without wrapping, clipping into the subject.
Fix: Set max-width: calc(100% - padding*2) on each .cap. Templates do this.
Had letterSpacing: "0.04em" as the “from” value in tl.fromTo(word, {opacity:0, letterSpacing: ".04em"}, {letterSpacing:"0"}, w.start). GSAP snaps the “from” state at w.start. The width of the word then changes over the tween → inline-block reflow → the .cap line box recomputes → “Some” visibly jumped between rows as “memories” entered.
Fix: Animate only transform properties (opacity, y, scale). NEVER animate letter-spacing, font-size, or filter:blur on .w spans. Templates follow this.
Fading both the group container AND each word independently multiplies: effective_opacity = container.opacity × word.opacity. Even with matched eases, the combined curve is nonlinear and feels like a snap-in around the 40% mark.
Fix: Keep container at fixed opacity: 1 during the group’s life (set via tl.set just before first word). Animate only per-word opacity. Templates do this.
overlay(white_text, black_bg) ≈ black. If part of the caption falls on a dark shadow / dark prop, the letters disappear. The first champion render had SHARP become ARP because “S” and “H” landed in the dark mic’s shadow.
Fix: For scenes with significant dark regions under the caption, use mix-blend-mode: screen (which preserves white on dark). wall-embed defaults to overlay (for mid-tone walls); corner-column-crown defaults to screen (for dark bookshelves).
Opposite problem: screen(white, bright) ≈ white regardless of the text — letters become invisible on bright backgrounds like sunlit windows or white walls.
Fix: If average luminance of the caption region exceeds 180/255, switch CSS to mix-blend-mode: normal with an opaque color. Sample the background pixels in the caption bbox across the clip to check.
If hyperframes renders at 30fps default but the matte was extracted at 24fps source rate, ffmpeg overlay gets a temporal mismatch — alpha from frame N is overlaid on bg content of some fractional frame ≠ N. Result: person’s current silhouette and matte silhouette are shifted by a few frames, so the occlusion lags or leads the body.
Fix: Always pass --fps to hyperframes render matching the source’s native FPS (usually 24). render-and-composite.sh reads fps from plan.json and passes it through.
We tried making foreground an alpha-channel WebM. Chromium’s support for VP9 alpha is inconsistent; VP8 alpha needs specific metadata the default ffmpeg encode doesn’t produce.
Fix: Don’t try to put the matte inside hyperframes. Do the overlay in post with ffmpeg against a PNG sequence. render-and-composite.sh does this.
Default .crown-plane in corner-column-crown is left: 0; right: 0; text-align: center. That mechanically centers the word at frame_width/2. If the subject sits right of center (Jobs at x=1100 in a 1920 frame) OR the word isn’t wide enough to poke both clean zones, the body eats 50-70% of the crown and it reads as “THE ___ S”.
Fix: Check subject center_x vs frame center before choosing crown strategy (see layout-heuristics.md § Crown placement). Either size the crown large enough to cross both clean zones, or shift it into the larger clean zone with a narrower width. Keep text-align: center within the (re-positioned) box, not the frame.
On landscape 1920×1080, a centered crown at 140px is only ~500px wide. If the subject is 700-900px wide, the crown vanishes behind them. Symptom: only “THE” and “S” visible for “THE BEATLES”.
Fix: For full-frame centered crown, font must be big enough that crown_width > subject_width + 400. See font-size table in typography-presets.md. Prefer 180-260px for landscape centered crowns.
Template defaults (66/78/92/140) assume a ~560px column. Expanding the plane to 700+ px without bumping fonts makes the text feel too small relative to the column.
Fix: Column width 600-760 → phrase ~108px, emph ~128px, crown ~220px centered. See table in typography-presets.md.
The matte is temporally smoothed (EMA in matte.cjs) but moderate motion only. Fast camera pans (vlog selfie) still produce frame-to-frame alpha jitter that shows as flickering caption edges.
Fix: SKILL.md decision gate lists handheld vlog as ⚠️ — degrade or refuse. Production version would add vidstabdetect compensation; v1 does not.
The human-matting model segments “foreground” as a single alpha. With two people, it captures both but they’re treated as one blob. Captions can’t go “behind person A but in front of person B”. If the scene has both speakers visible, the skill should split by shot boundaries first.
Fix: Either pre-split with PySceneDetect + per-shot planning, or refuse for ⚠️ multi-subject scenes.
Some sources (interviews, broadcast clips) cut to B-roll/archive footage mid-clip. The matte + caption layout assumes the subject is the same person throughout — a hard cut mid-render creates garbage (caption placement relative to old subject applied to a totally different shot).
Fix for v1: Probe the clip at 3-5 timestamps before accepting it. If any frame doesn’t contain the primary subject at the same position, trim the clip to end before the cut. Steve Jobs 60 Minutes cut to Beatles archival at t≈9s, was manually trimmed to 8.5s.
Fix for future: Run PySceneDetect as part of the admission gate; split into shots automatically.
TV archive clips often have pillarbox (black bars at sides) plus baked lower-third logos (“60 Overtime”) and date labels (“2003”). These are part of the video file, can’t be matted out, and shrink the usable frame area.
Fix: Detect black-bar margins in the first frame (scan rows/columns for all-black). Constrain caption safe-zone to [left_margin+pad, right_margin-pad, top_margin+pad, bottom_margin-pad]. For Jobs, safe content zone was x=280-1640 (not 0-1920). Layout positions must respect this.
Transcription is Whisper now, via transcribe.cjs (it wraps hyperframes transcribe — no API key). hyperframes init --video <mp4> may itself auto-write a transcript.json in hyperframes’ raw whisper shape (a flat word array, no top-level language_code). transcribe.cjs only treats a transcript as done when it’s ALREADY in our normalized schema ({ words: [...], language_code }); otherwise it (re-)runs Whisper and normalizes the flat word list into { words:[{text,start,end,type:"word"}], language_code }.
Fix / expectation: If you init via hyperframes first, expect transcribe.cjs to normalize that transcript into our schema. Don’t hand-leave a half-normalized file (e.g. our words shape but no language_code) — that’s the one state the skip-guard can misread.
If the original audio has a long silence, leaving the caption plane empty feels wrong mid-video. Either (a) let the prior caption linger, (b) show a title-card crown during the silence, or (c) cut the silence out of the clip.
Fix: During caption-grouping, detect silences > 3s. The skill should prompt the user about this, not silently proceed.
