HyperFrames Media Preprocessing

Three CLI commands that produce assets for compositions: tts (speech), transcribe (timestamps), and remove-background (transparent video). Each downloads a model on first run and caches it under ~/.cache/hyperframes/.

intent

use this skill when you need to preprocess media assets for hyperframes compositions. three commands cover the main flows: generate voiceover narration from text via kokoro tts, extract word-level timestamps from speech via whisper, or segment a person/subject from video/image for transparent overlays via u2net. each command is self-contained, downloads its own model on first use, and outputs artifacts you drop directly into the composition html. chain these (e.g., tts → transcribe → captions) to automate narration + caption timing.

inputs

• nodejs + npm installed (tested on node 16+)
• python 3.8+ for tts and transcribe commands
• system packages (non-english tts only): espeak-ng for phonemization. install via brew install espeak-ng (macos) or apt-get install espeak-ng (linux)
• pip packages: kokoro-onnx, soundfile (tts); openai-whisper or faster-whisper (transcribe); torch, torchvision, onnxruntime (remove-background)
• disk space: ~311 MB (kokoro model) + ~27 MB (voices) + ~168 MB (u2net weights) + variable space for outputs. all cached in ~/.cache/hyperframes/
• gpu optional: remove-background runs faster on cuda/mps; cpu fallback works (slower)
• audio/video inputs: mp3, wav, m4a, aac, flac (audio); mp4, mov, webm, mkv (video). image inputs: jpg, png for remove-background
• transcription inputs alternate formats: existing .srt/.vtt subtitle files, or json response blobs from openai/groq apis

procedure

command 1: text-to-speech (tts)

step 1: choose or infer the voice id based on content type and language.

• input: content description (product demo, tutorial, marketing, documentation, casual), target language
• output: voice id string, e.g. af_heart, bf_emma, jf_alpha
• reference the voice selection table below. if language matches voice prefix auto-detect, no --lang override needed. if you need a stylized accent or the text language doesn't match voice locale, use --lang to override

step 2: prepare input text.

• input: prose (short) or path to .txt file (long scripts)
• output: text string or file path
• for scripts longer than ~5 minutes of speech, split into segments and run multiple tts commands

step 3: set speech speed if non-default.

• input: content type (tutorial vs. upbeat intro vs. accessibility)
• output: --speed flag value (0.7-1.5 range)
• default is 1.0. use 0.7-0.8 for complex/accessibility, 1.1-1.2 for intros, 1.5+ rarely appropriate

step 4: run tts command.

• input: text/file path, voice id, output filename, optional speed
• output: .wav file at specified path
• command: npx hyperframes tts "<text>" --voice <id> --output <file.wav> or npx hyperframes tts <script.txt> --voice <id> --output <file.wav> --speed <0.7-1.5>
• on first run, downloads ~311 MB kokoro model and ~27 MB voice pack to ~/.cache/hyperframes/tts/. subsequent runs use cache
• if espeak-ng not installed and --lang is non-english, command fails with instruction to install it

step 5: verify audio output.

• input: generated .wav file
• output: playback confirmation
• listen for correct language, voice tone, and speech rate. if mismatch, re-run step 1-4 with corrected voice or speed

command 2: transcription (transcribe)

step 1: determine the language of the audio.

• input: audio/video file or filename hint
• output: language code (e.g. es, en, ja, fr-fr) or "unknown"
• listen to a short clip or check metadata. if unknown, skip --language flag and whisper auto-detects

step 2: select whisper model size.

• input: language known?, audio quality (clear/noisy), content type (quick preview vs. production)
• output: model name string
• default is small (466 MB, moderate speed). use tiny (75 MB) for preview/testing, base for short clips, medium (1.5 GB) for noisy audio/music, large-v3 (3.1 GB) for production. never suffix with .en unless you explicitly know audio is english and you want whisper to translate non-english into english (destructive)

step 3: run transcribe command.

• input: audio/video file path or subtitle file (.srt/.vtt) or json response file
• output: transcript.json in current directory
• command: npx hyperframes transcribe <file> (auto-detects language, uses model small), or npx hyperframes transcribe <file> --model <size> --language <code> (explicit language + model)
• for .srt/.vtt import, just pass the file path , command normalizes to json
• for openai/groq api json blobs, pass the file path , command extracts words + timestamps
• on first run, downloads the model (~75 MB to ~3.1 GB depending on size) to ~/.cache/hyperframes/transcribe/models/. subsequent runs use cache
• processing time scales with audio length and model size. small on 10 min audio takes ~30 sec; large-v3 takes ~3 min

step 4: inspect transcript.json for accuracy.

• input: generated transcript.json
• output: validation pass/fail
• open the json and scan word boundaries. if timing is off or words are garbled, re-run step 2-3 with a larger model (medium or large-v3)
• see hyperframes/references/transcript-guide.md for caption quality checks and cleaning js

step 5: drop transcript.json into composition.

• input: transcript.json
• output: file placed in project directory
• reference it from composition html for caption sync (see hyperframes skill)

command 3: background removal (remove-background)

step 1: prepare the source video or image.

• input: .mp4 (or .mov, .webm, .mkv) or .jpg/.png
• output: file path and format confirmation
• for video, clip should be 5-60 sec (longer clips mean longer processing). for image, any resolution. ensure subject (person/avatar) is reasonably clear and fills ~20-50% of frame for best segmentation

step 2: choose output format and layer strategy.

• input: use case (overlay on different scene vs. text-behind-subject), whether you need both layers
• output: output format (.webm, .mov, or .png) and --background-output flag (yes/no)
• default is .webm (vp9 + alpha, native browser decode). use .mov (prores 4444) only if you're editing in davinci/premiere/fcp. use .png only for single-image cutouts. if you want both the subject layer (opaque subject, transparent background) and the plate layer (transparent subject, opaque background), add --background-output <plate.webm> flag

step 3: set quality preset if non-default.

• input: iteration phase (rough draft vs. master delivery), file size constraints
• output: --quality flag value
• default is balanced (crf 18, good balance). use fast (crf 30) for iteration and smaller files, best (crf 12) for final delivery and tightest color match. segmentation quality is fixed; crf only affects vp9 encode

step 4: optionally specify device.

• input: gpu availability (cuda/mps) vs. cpu-only
• output: --device flag value
• default is auto-detect (cuda if available, else mps on mac, else cpu). explicitly pass --device cpu to force cpu (slower but no cuda/mps dependency)

step 5: run remove-background command.

• input: source file path, output path, format, optional flags
• output: .webm (or .mov/.png) at specified path, optionally a second plate file
• command: npx hyperframes remove-background <source> -o <output.webm> (basic cutout), or npx hyperframes remove-background <source> -o <transparent.webm> --background-output <plate.webm> --quality best (both layers, high quality)
• on first run, downloads ~168 MB u2net weights to ~/.cache/hyperframes/background-removal/models/. subsequent runs use cache
• processing time: ~30 sec (tiny clip) to ~5 min (60 sec hd video) on gpu; 2-3x slower on cpu

step 6: verify segmentation and composite structure.

• input: generated cutout and optional plate .webm files
• output: visual inspection pass/fail
• open cutout in a player with alpha support (chrome, ffplay) and inspect silhouette edges for artifacts. if edges are jagged or subject is over-trimmed, re-run with --quality best. if segmentation missed a limb or include stray background, re-run with a better-lit source or crop the source to focus the frame
• if using --background-output plate, verify the subject region in plate is fully transparent and the surroundings are opaque. plan what content (text, graphics, solid color) will sit between the plate and cutout layers

step 7: drop cutout (and plate) into composition.

• input: .webm file(s)
• output: file(s) placed in project directory, referenced in composition html
• see compositing patterns section below for html structure and z-index / opacity rules

command 4: chained workflow (tts → transcribe → captions)

step 1-5: run tts command (see command 1, steps 1-5).

• output: narration.wav

step 6: run transcribe command on the generated narration.

• input: narration.wav
• command: npx hyperframes transcribe narration.wav
• output: transcript.json with precise word-level timestamps extracted from the generated speech
• no need to specify --language or --model if default small + english auto-detect works (it will for english voices)

step 7: use transcript.json for caption timing in composition.

• input: transcript.json, composition html
• output: captions synced to narration playback
• word boundaries from whisper are accurate to within ~50ms, so captions will match delivery without hand-tuning

decision points

tts language / voice mismatch:

• if text language matches voice prefix (e.g., spanish text + ef_dora voice), no --lang needed. whisper auto-detects from voice id
• if text language does NOT match voice locale (e.g., english text + jf_alpha voice, or stylized accent needed), pass --lang to override. valid codes: en-us, en-gb, es, fr-fr, hi, it, pt-br, ja, zh
• if espeak-ng not installed and --lang is non-english, tts fails. install espeak-ng and retry

transcribe language rule (non-negotiable):

• if language is known AND non-english: use --model small --language <code> (no .en suffix). .en models translate non-english audio into english (destructive)
• if language is known AND english: use --model small.en (optional; small without .en also auto-detects english correctly)
• if language is unknown: use --model small (no .en, no --language). whisper auto-detects
• default model is small, not small.en. never guess .en if unsure

transcribe input format:

• if input is .srt or .vtt, command normalizes to transcript.json
• if input is openai/groq json response blob, command extracts words + timestamps
• if input is audio/video, whisper transcribes and extracts timestamps

remove-background output format:

• for overlay on different scene: use .webm (vp9 + alpha, native browser support, default)
• for editing in davinci/premiere/fcp: use .mov (prores 4444, larger file, lossless)
• for single-image cutout (still subject over backdrop): use .png

remove-background layer strategy:

• if compositing text/graphics between subject and background: use --background-output <plate> to emit both cutout and plate. subject layer goes on top (z-index 3), plate on bottom (z-index 1), text/graphics in middle (z-index 2)
• if subject overlays unrelated scene: just use cutout, ignore plate
• if goal is to remove subject and show clean room alone: do NOT use this command. inpainting (LaMa, ProPainter, E2FGVI) needed instead

remove-background quality preset:

• if iterating or file size constrained: use --quality fast (crf 30)
• if general use or default: use --quality balanced (crf 18, default)
• if text-behind-subject or master delivery: use --quality best (crf 12) to minimize color shift / edge halo when cutout overlays re-encoded version of same source

remove-background device selection:

• if cuda/mps gpu available and no constraints: omit --device flag (auto-detect)
• if cuda/mps unavailable or want to force cpu: pass --device cpu (slower, 2-3x latency)

remove-background with multiple layers (text-behind-subject):

• wrap cutout video in non-timed <div> and animate wrapper opacity, not video opacity. framework forces opacity:1 on active clips (elements with data-start/data-duration), so video-level opacity is silently overridden
• both videos (source mp4 and cutout webm) must use data-start="0" and data-media-start="0" so they decode in sync. late-mounting cutout introduces seek warm-up that lands a frame off the source , visible as misalignment
• flip wrapper opacity at cut point using gsap or css animation

output contract

tts outputs:

• single .wav file at path specified in --output flag
• format: pcm 24-bit 44.1 khz mono (compatible with whisper and composition <audio> elements)
• file size: ~500 kb per minute of speech (depends on bitrate, voice, speed)
• located in current working directory or absolute path if specified

transcribe outputs:

• single transcript.json in current working directory
• schema: flat array of word objects, each with required fields text, start, end (float seconds) and optional id field (w0, w1, ..., for caption override references)
• example: [{"id":"w0","text":"Hello","start":0.0,"end":0.5},{"id":"w1","text":"world.","start":0.6,"end":1.2}]
• timing precision: +/- 50ms word boundaries for small/medium models, +/- 20ms for large-v3
• for multi-speaker audio, all words in one flat array (no speaker attribution)

remove-background outputs:

• primary: .webm (vp9 + alpha, default) or .mov (prores 4444) or .png (single image)
• vp9 webm: h.264 rgb + vp9 alpha channel, native decode in chrome/firefox/edge, plays in <video> elements. file size: ~2-5 mb per minute of hd 1080p video depending on quality preset
• prores 4444: prores rgb + uncompressed 8-bit alpha, for editing suites. file size: ~200-400 mb per minute of hd 1080p video (lossless, no quality presets)
• png: single frame, lossless rgba, any resolution. file size: ~2-10 mb depending on resolution and compression
• optional secondary (if --background-output flag): plate file (inverse alpha, same format as primary). when both exist, sum of both files is ~1.3x size of single cutout
• located in path specified in -o flag (or current directory if relative path)

remove-background segmentation quality:

• fixed (not affected by --quality preset). u2net_human_seg trained on portrait photography, works best on people. objects, animals, or non-portrait framing may require manual refinement
• edge precision: typically +/- 2-5 pixels depending on lighting, motion blur, and subject contrast. over-segmentation (cutting into subject) more common than under-segmentation (leaving background) in high-contrast hair/fur

outcome signal

tts success:

• .wav file exists at specified output path
• play file in any audio player; narration is intelligible, voice tone matches intent (warm vs. energetic vs. formal), speech rate feels natural and matches content type
• duration matches input text (rough: ~150 words per minute at 1.0 speed)
• no audio clicks, pops, or dropout (rare, only on very long scripts or system resource constraints)

transcribe success:

• transcript.json exists in current directory
• file is valid json; array of word objects with text, start, end fields all populated
• open transcript in text editor and spot-check ~10 random words against audio. word boundaries should align within ~100ms of spoken word start/end
• if transcription accuracy is poor (garbled words, missing words