Phy Video Bgm

intent

use gemini's video understanding to analyze a video's mood, brand positioning, and emotional journey, then generate matching background music via fal.ai lyria2, mix it in with ffmpeg (with optional speed adjustment), and output a final video with bgm. use this skill when you need to add contextually appropriate background music to a video without manually composing or licensing tracks.

inputs

required parameters:

• path: file path to input video (mp4, mov, or other ffmpeg-compatible format)

optional parameters:

• --speed: speed multiplier for video playback (default 1.0, e.g. 1.1 = 10% faster, 0.9 = 10% slower)
• --volume: bgm volume multiplier (default 5.0, since lyria2 output is quiet)
• --fade-in: fade-in duration in seconds (default 1.5)
• --fade-out: fade-out duration in seconds (default 3.0)
• --style: override music style and skip gemini analysis (optional string, e.g. "upbeat modern pop, synth pads, 100 bpm")

external connections:

• gemini api: requires GOOGLE_GENAI_API_KEY env var. needs video understanding capability (gemini 2.0 flash or later). oauth not required, api key auth only.
• fal.ai lyria2: requires FAL_API_KEY env var. music generation model. https://fal.run/fal-ai/lyria2. api key auth only.
• ffmpeg: system binary for video encoding/mixing. must be installed and in system path.

system dependencies:

• python 3.8+
• pip packages: google-generativeai, httpx
• ffmpeg (system install, check with ffmpeg -version)
• ffprobe (comes with ffmpeg)

procedure

step 1: validate inputs and set up

inputs: video file path, optional speed/volume/fade/style parameters.

• check that input video file exists and is readable.
• verify GOOGLE_GENAI_API_KEY and FAL_API_KEY env vars are set.
• verify ffmpeg and ffprobe are in system path.
• if any check fails, exit with clear error message.

outputs: validated file path, parsed parameters (speed, volume, fade-in, fade-out, style override).

step 2: analyze video with gemini (unless style override provided)

inputs: video file path, gemini api key.

• configure gemini client with api key.
• upload video to gemini 2.0 flash using file upload api.
• send music supervisor prompt: ask gemini to analyze (1) brand positioning, (2) emotional journey at key moments, (3) real commercial music references (e.g. Four Seasons, Apple, Nike style), (4) energy level (contemplative vs forward momentum), (5) tempo/instruments/production style. request a single 2-3 sentence music generation prompt at the end, prefixed with MUSIC_PROMPT:.
• wait for gemini response (may take 10-30s depending on video length and api load).
• parse response to extract the line starting with MUSIC_PROMPT:. extract everything after that prefix as the music prompt string.

outputs: music prompt string extracted from gemini response. if gemini response is malformed or times out, exit with error.

step 3: append constraints to music prompt

inputs: music prompt string from step 2 (or from style override).

• append this constraint string to the prompt: "no vocals, no drums, no percussion hits, no sound effects."

outputs: final music prompt ready for lyria2.

step 4: generate bgm via fal.ai lyria2

inputs: final music prompt, fal api key, timeout 120 seconds.

• make http post to https://fal.run/fal-ai/lyria2 with json body {"prompt": music_prompt} and header Authorization: Key {FAL_API_KEY}.
• wait for response (may take 30-60s).
• extract audio url from response json at response["audio"]["url"].
• download audio file from url (lyria2 outputs ~32 seconds of 48khz stereo wav).
• save bgm wav to disk next to input video with name pattern {input_stem}_bgm.wav.

outputs: bgm wav file on disk. if lyria2 times out or rate-limits, retry once after 5 second delay; if second attempt fails, exit with error.

step 5: speed up video and strip existing audio (if speed != 1.0)

inputs: input video path, speed multiplier, output path.

• if speed == 1.0, skip to step 6.
• if speed != 1.0, run ffmpeg: ffmpeg -y -i INPUT.mp4 -filter:v "setpts=PTS/{speed}" -an -c:v libx264 -preset medium -crf 18 OUTPUT_speedup.mp4.
• this re-encodes the video at new speed and removes any existing audio track (-an flag).
• save output as {input_stem}_speedup.mp4.

outputs: sped-up video file (or original video if speed == 1.0). if ffmpeg encode fails or times out, exit with error.

step 6: get duration of (possibly sped-up) video

inputs: video file path (speedup or original).

• run ffprobe -v quiet -show_entries format=duration -of csv=p=0 {video_file} to get duration in seconds.
• store as float variable duration.

outputs: duration in seconds. if ffprobe fails, exit with error.

step 7: mix bgm with video, apply volume/fade, trim to video length

inputs: video file (speedup or original), bgm wav file, volume multiplier, fade-in duration, fade-out duration, output path, video duration.

• run ffmpeg: ffmpeg -y -i {video_file} -i {bgm_wav} -filter_complex "[1:a]volume={volume},atrim=0:{duration},afade=t=in:st=0:d={fade_in},afade=t=out:st={duration-fade_out}:d={fade_out}[a]" -map 0:v -map "[a]" -c:v copy -c:a aac -b:a 192k -shortest {output_file}.
• this applies volume boost to bgm, trims bgm to video duration, applies fade-in and fade-out, mixes bgm with video, and encodes audio as aac 192k.
• save output as {input_stem}_final.mp4.

outputs: final video with bgm mixed in. if ffmpeg mix fails, exit with error.

step 8: report completion and open files

inputs: final video path, bgm wav path.

• print completion message with paths to final video and bgm wav.
• open final video in default video player (platform-specific: open on macos, xdg-open on linux).
• open bgm wav in default audio player so user can review music separately.

outputs: user can see/hear result. if open fails, log warning but do not exit.

decision points

• if --style parameter is provided: skip step 2 (gemini analysis) and use the provided style string directly as the music prompt. jump to step 3 (append constraints).

• if speed == 1.0: skip step 5 (re-encode video) and use original video directly in step 7. this saves encoding time and quality loss.

• if video file does not exist or is unreadable: exit with error message stating file path and suggesting user check path.

• if GOOGLE_GENAI_API_KEY or FAL_API_KEY env vars are missing: exit with error message listing which keys are missing and instructions to set them.

• if ffmpeg or ffprobe are not in system path: exit with error message and suggest user install ffmpeg (brew install ffmpeg on macos, apt-get install ffmpeg on ubuntu).

• if gemini response does not contain MUSIC_PROMPT: line: exit with error stating gemini response was malformed. suggest user try again or provide explicit style override.

• if lyria2 api returns 429 (rate limit) on first attempt: retry once after 5 second exponential backoff. if second attempt also rate limits, exit with error.

• if lyria2 api returns 401 (auth failure): exit with error stating FAL_API_KEY is invalid or expired.

• if ffmpeg encode takes > 5 minutes for a single video: log warning and ask user if they want to cancel. continue if user approves.

• if video is shorter than fade-out duration: adjust fade-out to (duration - 0.5) to ensure fade does not exceed video length.

• if bgm file is empty or corrupt after download: exit with error and suggest user retry lyria2 generation.

output contract

success is defined by:

• final video file at path {input_stem}_final.mp4 (or same directory as input). format is mp4 with h.264 video, aac 192k audio, video duration unchanged (or sped up/slowed down as requested), bgm mixed with fade-in/out applied.

• bgm wav file at path {input_stem}_bgm.wav. format is wav, 48khz stereo, ~32 seconds duration (generated by lyria2).

• intermediate files (if speed != 1.0): {input_stem}_speedup.mp4 (sped-up video with no audio).

• no output if any validation or generation step fails. original video and any partial intermediate files remain untouched.

• file naming: all output files saved in same directory as input video.

outcome signal

• user sees completion message printed to stdout listing paths to final video and bgm wav file.
• final video file opens automatically in default video player and plays with bgm mixed in, volume boosted, and fades applied.
• bgm wav file opens automatically in default audio player so user can evaluate music quality separately.
• user can verify