👄 Lipsync , Pro Pack on RunComfy

intent

sync a face's mouth to an audio track using the runcomfy CLI. the skill classifies your intent (portrait still + audio for avatar-style talking head, source video + audio for mouth-swap onto existing footage, or script-only for in-pass audio generation), routes to the right model (Sync Labs sync v2/Pro, ByteDance OmniHuman, Kling lipsync, or Creatify), and executes the invoke. use this when you need to drive a face's mouth from a voiceover, dub a video in another language, create a talking-head avatar from a portrait, or generate a character video with synced speech from a script.

inputs

external connection: runcomfy CLI

• install: npm i -g @runcomfy/cli or use npx -y @runcomfy/cli --version for verification
• authentication: set RUNCOMFY_TOKEN env var (from runcomfy login which writes to ~/.config/runcomfy/token.json with mode 0600) or run runcomfy login interactively
• scope: read/write access to the Model API (implicit in standard runcomfy auth)
• network: outbound to model-api.runcomfy.net, *.runcomfy.net, *.runcomfy.com only

context & parameters (user-provided)

• intent classification inputs (user states one of these):
  • "i have a video + audio to sync" (source video + audio route)
  • "make an avatar talk from a portrait + voiceover" (portrait still + audio route)
  • "i have a script but no audio file" (generate-and-sync route)
• asset URLs (HTTPS, user-curated):
  • video_url: source video file (mp4 preferred; Sync Labs, Kling a2v, Creatify routes)
  • image_url: portrait still (jpg/png; OmniHuman route)
  • audio_url: voiceover MP3 or WAV (Sync Labs, Kling a2v, OmniHuman, Wan 2.7 routes)
  • script: text prompt (Kling t2v, HappyHorse t2v routes; generates audio in-pass)
• optional: quality/budget tier (user preference):
  • premium tier: Sync Labs sync v2 Pro (higher cost, best mouth fidelity)
  • standard tier: Sync Labs sync v2, Kling, Creatify (lower cost, acceptable quality)
  • open-weights: Wan 2.7, HappyHorse 1.0 (lowest cost, full scene control)
• optional: output directory (default: ./out; where downloaded files land)

edge cases to surface

• audio/video duration mismatch (drift if audio >> video or video >> audio; trim or extend first)
• audio quality (music bed + voice = poor sync; isolate voice stem if needed)
• source video frame rate (Sync Labs works best at 24-30 fps; unusual fps may degrade sync)
• portrait framing for OmniHuman (head-and-shoulders works best; full-body or extreme crops degrade results)
• token expiry (if RUNCOMFY_TOKEN is stale, runcomfy login refreshes it)
• network timeout (default 5-10 min for large videos; monitor polling)
• rate limits (429 errors from upstream; retry with backoff)
• empty result set (job completes but no output video; check source asset URLs and schema)

procedure

1. classify user intent

   • input: user's stated goal ("sync this video to audio", "make a talking avatar", "write a script and generate video with speech")
   • logic: map to one of three routes (source video + audio, portrait still + audio, script-only)
   • output: confirmed route name (e.g., "Sync Labs sync v2 Pro", "OmniHuman", "Kling text-to-video")

2. validate runcomfy CLI availability and auth

   • input: shell environment
   • run: runcomfy --version to confirm CLI is installed
   • run: runcomfy whoami to confirm RUNCOMFY_TOKEN is set and valid
   • output: CLI version string and authenticated user ID, or error (exit code 77 if not signed in)
   • if not signed in: direct user to runcomfy login or set RUNCOMFY_TOKEN env var

3. acquire and validate asset URLs

   • input: user's source video, portrait, audio, or script text
   • validate video_url / image_url / audio_url are HTTPS and reachable (HEAD request to confirm)
   • validate audio_url points to MP3 or WAV (check Content-Type header)
   • validate audio/video duration match for sync routes (log warning if duration ratio > 1.2x or < 0.8x)
   • output: list of validated URLs and duration estimates
   • on URL unreachable or wrong content type: fail with user-facing error, do not invoke runcomfy

4. select model endpoint based on route and quality tier

   • input: route from step 1, user's quality/budget preference (if stated)
   • logic: apply decision tree (see decision points section)
   • output: model slug (e.g., sync/sync/lipsync/v2/pro, bytedance/omnihuman/api, kling/lipsync/audio-to-video)

5. build JSON input schema for chosen model

   • input: validated URLs from step 3, model slug from step 4
   • construct JSON object matching the model's documented schema:
     • Sync Labs: {"video_url": "...", "audio_url": "..."}
     • OmniHuman: {"image_url": "...", "audio_url": "..."}
     • Kling a2v: {"video_url": "...", "audio_url": "..."}
     • Kling t2v: {"prompt": "..."}
     • HappyHorse t2v: {"prompt": "... says clearly: \"<script>\" ..."}
   • output: JSON string ready for --input flag

6. invoke runcomfy run with model slug and input JSON

   • input: model slug, JSON input, output directory (default ./out)
   • run: runcomfy run <model-slug> --input '<json>' --output-dir ./out
   • capture stdout/stderr and exit code
   • output: runcomfy job ID (from stdout), polling status

7. poll runcomfy job status and fetch results

   • input: job ID from step 6
   • CLI auto-polls; monitor for completion (typically 1-5 min for OmniHuman, 2-10 min for Sync Labs Pro depending on video length)
   • on job success (exit 0): CLI auto-downloads .runcomfy.net / .runcomfy.com URLs into --output-dir
   • output: local file path(s) of downloaded result video(s)
   • edge case: job status 429 or timeout (exit 75); user may retry without re-uploading assets

8. validate output and confirm sync quality

   • input: downloaded result video from step 7
   • spot-check: play first 10 sec of video, confirm mouth movements align with audio
   • output: user-facing confirmation message with result file path and basic validation status
   • on obvious sync failure (mouth closed entire clip, or completely misaligned): flag as degraded quality; suggest checking source audio quality or retrying with OmniHuman if route was Sync Labs

decision points

route selection tree

• if user provides source video + audio:

  • ask: "what's your budget tier?"
    • if premium (cost acceptable, hero-quality dub needed): use Sync Labs sync v2 Pro (sync/sync/lipsync/v2/pro)
    • if standard (cost-sensitive, acceptable quality): use Sync Labs sync v2 (sync/sync/lipsync/v2)
    • if alternative preferred: offer Kling lipsync a2v (kling/lipsync/audio-to-video) or Creatify lipsync (creatify/lipsync)
  • proceed to step 3 with video_url and audio_url

• else if user provides portrait still + audio:

  • ask: "do you want simple avatar (just the portrait speaks) or full scene control?"
    • if simple avatar: use OmniHuman (bytedance/omnihuman/api)
    • if full scene (different background, multiple people, props): use Wan 2.7 (wan-ai/wan-2-7/text-to-video) with audio_url field and visual prompt
  • proceed to step 3 with image_url and audio_url (or prompt + audio_url for Wan)

• else if user provides script text only (no audio file):

  • ask: "do you want in-pass audio generation + mouth sync?"
    • if yes and avatar-style: Kling lipsync t2v (kling/lipsync/text-to-video) takes prompt, generates speech and video
    • if yes and high quality overall: HappyHorse 1.0 t2v (happyhorse/happyhorse-1-0/text-to-video) with prompt containing says clearly: "<dialogue>", generates audio + video
    • if no / user wants to record audio separately: exit skill, direct user to record voiceover first, then re-run with portrait + audio
  • proceed to step 3 with prompt (no URL validation needed for text prompts)

• on audio/video duration mismatch (ratio > 1.2x or < 0.8x):

  • log warning: "audio and video duration differ significantly; sync may drift. consider trimming audio or extending video first."
  • ask user: "proceed anyway, or adjust?"
    • if proceed: continue to step 5 with note in final report
    • if adjust: exit skill, direct user to edit assets

• on token expiry (runcomfy whoami returns 401 or exit 77):

  • offer two paths:
    • if interactive terminal: suggest runcomfy login and re-run
    • if non-interactive (CI/container): suggest setting RUNCOMFY_TOKEN env var and re-run

• on job failure (exit 65 schema mismatch, 69 upstream 5xx, 75 timeout/429):

  • exit 65: re-check JSON schema against model docs; user likely passed wrong field names or types
  • exit 69: upstream API error; suggest retry after 60 sec
  • exit 75: timeout or rate-limit; suggest retry with exponential backoff; note job may still complete server-side

• on obvious sync failure after download:

  • if Sync Labs route: suggest source audio is low quality (music bed, background noise); direct user to isolate voice stem
  • if OmniHuman route: suggest portrait framing issue (too close, extreme angle, or not head-shoulders); direct user to re-frame portrait
  • if Kling t2v route: suggest script did not map to dialogue; direct user to use says clearly: "<exact phrase>" syntax

output contract

success state

• exit code 0 from runcomfy run
• result video file downloaded to --output-dir (default ./out)
• file format: .mp4 (H.264 or H.265 codec, 24-30 fps recommended)
• file size: < 2 GiB (CLI enforces cap)
• metadata: video duration matches source (for Sync Labs / Kling a2v) or is newly generated (for OmniHuman / Kling t2v / HappyHorse)
• mouth sync: first 10 sec of video shows mouth movements roughly aligned with audio (no perfect pixel-lock expected; audio leading slightly is normal)

failure states

• exit code 64: bad CLI args; re-check model slug and --input JSON syntax
• exit code 65: input JSON schema mismatch; review model's documented fields and types on runcomfy.com/models/
• exit code 69: upstream API error (5xx); message includes HTTP status; user may retry
• exit code 75: timeout or rate-limit (429); job may still complete server-side; user may retry with backoff
• exit code 77: not authenticated; RUNCOMFY_TOKEN not set, invalid, or expired
• no output file despite exit 0: upstream job succeeded but generated no video; check source asset URLs and try alternative model

user-facing output

• confirmation message: "synced to using . result: "
• quality note: "mouth sync looks good" (after spot-check) or "sync degraded; check source audio quality"
• file location: absolute path to result video in --output-dir

outcome signal

the user knows the skill worked when:

1. CLI runs without error (exit code 0) and logs a job ID
2. result video downloads to the specified output directory (./out by default)
3. video plays (can open in VLC, ffplay, or browser)
4. mouth movements sync to audio (user scrubs through first 10-30 sec and sees lips move roughly in time with speech; audio leading slightly is normal and acceptable)
5. source identity preserved (for Sync Labs and Kling a2v: original face/body intact; for OmniHuman: portrait-derived avatar has the source subject's likeness)
6. no warnings in final report (or warnings are benign: e.g., "audio is 2 sec longer than video, sync may drift at end")

if the user cannot play the result video, mouth sync is visibly misaligned (audio completely out of sync), or identity is distorted, the skill failed; review error message and retry with adjusted inputs or a different model tier.

full reference: model routes

Route 1: Sync Labs sync v2 / Pro (source video + audio)

model: sync/sync/lipsync/v2/pro (premium) or sync/sync/lipsync/v2 (standard) when to use: mouth-swap onto existing video (preserves original camera, lighting, body pose; only replaces mouth motion) quality: state-of-the-art fidelity (Pro > standard) cost: Pro > standard schema:

{
  "video_url": "https://...",
  "audio_url": "https://..."
}

tips:

• source video provides everything except the mouth. camera, lighting, background, body pose all preserved.
• audio quality drives mouth quality. clean voiceover (no music bed) syncs better. isolate voice stem if needed.
• match audio length to video length. significant mismatch leads to drift at the end. trim audio or extend video first.
• works best at 24-30 fps. unusual frame rates may degrade sync.

links: sync v2 Pro · sync v2

Route 2: OmniHuman (portrait still + audio)

model: bytedance/omnihuman/api when to use: avatar-style talking head from a single portrait (no separate audio recording required if you generate audio separately) quality: good for UGC, voiceover, virtual presenter use cases cost: low-to-mid schema:

{
  "image_url": "https://...",
  "audio_url": "https://..."
}

tips:

• portrait framing works best: head-and-shoulders or upper body.
• no prompt needed. the model derives everything from image + audio.
• gesture and body motion are generated; avatar may move beyond the original portrait frame.
• all dubs of the same portrait preserve identity across languages.

links: omnihuman

Route 3: Kling Lipsync (variants)

audio-to-video variant:

• model: kling/lipsync/audio-to-video
• when to use: sync mouth onto source video (Kling alternative to Sync Labs)
• quality: good; slightly behind Sync Labs Pro in fidelity
• schema:

  {
    "video_url": "https://...",
    "audio_url": "https://..."
  }
  

text-to-video variant:

• model: kling/lipsync/text-to-video
• when to use: generate video + synced speech from script only (no audio file needed)
• quality: good overall quality, in-pass audio generation
• schema:

  {
    "prompt": "a person says clearly: \"hello world\""
  }
  

links: Kling a2v · Kling t2v

Route 4