How do I take a raw source video plus an instruction markdown file and generate a set of cinematic b-roll clips with the Runway ML API, guided by the style and shot direction in the instructions

the agent that answers this

cinematic b-roll generator. Take a raw source video plus an instruction markdown file and generate a set of cinematic b-roll clips with the Runway ML API, guided by the style and shot direction in the instructions.

• Step 1tool

  Preflight before any work: verify ffmpeg is installed (`ffmpeg -version`) and install it idempotently if missing (`brew install ffmpeg` on macOS or `apt-get install -y ffmpeg` on Linux); verify Runway ML API access and credits before the first paid call, and if credentials are missing stop with the exact signup link and a cheaper Seedance-via-HeyGen alternative.

  Your model fills this step

  • No integration? If Runway ML access is missing, hold the run as needs-input before any paid generation begins.
• Step 2

  Read and parse the instruction markdown file into a structured brief containing cinematic style and mood, an ordered shot list with per-shot direction, aspect ratio, target clip count and duration, and any Runway model preference.

  Your model fills this step
• Step 3tool

  Ingest the raw source video and analyze it with ffmpeg: probe duration, resolution and fps, then extract representative keyframes or scene stills at the moments where the brief calls for b-roll.

  Video and audio processing with FFmpeg. Use for format conversion, resizing, compression, audio extraction, and preparing assets for Remotion. Triggers include…

  FFmpeg for Video Production FFmpeg is the essential tool for video/audio processing. This skill covers common operations for Remotion video projects. Quick Reference GIF to MP4 (Remotion-compatible) ffmpeg -i input.gif -movflags faststart…

  Full skill: ffmpeg

  • No integration? If ffmpeg is unavailable, sample stills at even intervals across the video and proceed with those.
  • No integration? If no usable frames can be extracted, fall back to text-to-video from the brief's shot descriptions only.
• Step 4

  For each shot in the brief, craft a detailed cinematic generation prompt covering camera movement, lens, lighting, mood, subject motion, scene continuity, and physical realism, paired with the matching seed keyframe.

  Seedance2 Video Generation Prompt Engineering — multimodal reference system, cinematic camera language, audio-video sync, and scene-by-scene prompt patterns...

  --- name: seedance-2-prompt description: Seedance2 Video Generation Prompt Engineering — multimodal reference system, cinematic camera language, audio-video sync, and scene-by-scene prompt patterns for ByteDance's Seedance2 model. license:…

  Full skill: Seedance2 Prompt Engineering
• Step 5tool

  Call the Runway ML API to generate one cinematic b-roll clip for each shot prompt using image-to-video from the seed frame or text-to-video when no frame exists; poll each task to completion and download the resulting MP4.

  Generate and edit video with Runway through RunAPI. Use when the user asks an agent to create, edit, or transform video with Runway. Default to the RunAPI CL...

  --- name: runway description: Generate and edit video with Runway through RunAPI. Use when the user asks an agent to create, edit, or transform video with Runway. Default to the RunAPI CLI for one-off generation; use SDKs only when the…

  Full skill: runway

  • No integration? If RUNWAYML_API_SECRET is not set, hold the run as needs-input asking the user to add their Runway API key, then resume.
  • No integration? If a generation fails or is moderated, retry once with a softened prompt and skip it after that, recording the failure in the manifest.
• Step 6

  Assemble the generated clips into an output bundle: name every clip for its shot label and write a manifest mapping it to the prompt, seed frame, source moment, Runway task id, and cost.

  Your model fills this step
• Step 7decision

  Quality gate: adversarially judge the deliverable from the lenses of a domain expert, a hard skeptic, and the end user, then WATCH EVERY GENERATED B-ROLL CLIP FROM START TO FINISH at normal speed and scrub it frame-by-frame around suspicious moments; do not judge from the manifest or isolated thumbnails alone. Also use ffmpeg to extract the first and last frame plus samples about every 0.5 seconds as a systematic coverage check. Inspect spatial anatomy and temporal continuity for genuine AI-generation mistakes, including extra/double/missing/fused hands, fingers or limbs; warped or morphing faces and bodies; duplicated, melting, disappearing, teleporting or identity-changing objects; broken reflections, shadows or physics; flicker, sudden texture/background changes, jumpy motion, continuity breaks, and unnatural or physically impossible transitions or camera moves. Record each defect with clip name, timestamp, severity, and evidence in the manifest. Reject any clip with a visible genuine defect: retry generation once with a corrected prompt or seed, then watch the replacement end-to-end and re-run the same checks. If it still fails, mark it hold-for-regenerate and exclude it from approved handoff. Only clips that pass both full-motion viewing and sampled-frame inspection may be marked approved; fix clear problems in place and state any blocker before handoff.

  Decision step
• Step 8decision

  Present the b-roll bundle and manifest for user review and approval before final handoff, clearly separating approved clips from any shots that were skipped, failed the full-motion visual defect check, or remain held for regeneration.

  Decision step

common questions

1. v3Jul 16manual

   Step 7 now watches every clip end-to-end, checks frames for anatomy and temporal defects, retries once, and excludes failed clips.

2. v2Jul 16manual

   rebound step 5 (gap -> clawhub/runapi-runway)

3. v1Jul 16generated

   auto-generated from "Take a raw source video plus an instruction markdown file and generate a set of "

Agents are alive: every change is a version, and a run can propose improvements that get reviewed and applied.

related agents

• creator

  How do I read a video script, identify each shot's spoken content, and generate specific generic stock B-roll direction per shot — constrained for a HeyGen AI twin talking head that cannot move or act, with twin filling lower frame and B-roll playing in the top half or as background

  B-Roll Director for AI Twin Videos

• dailycreator

  How do I every day, run a multi-persona debate over the day's research to pick the best reel angle, then produce the reel brief and the IG caption, ready for your avatar render

  Daily IG reel: debate and brief

• dailycreator

  How do I produce my daily on-brand vertical IG reel end to end — research the day's angle, write the avatar script, generate the avatar video, render it with captions, and prep the caption + guide link, held for my approval before posting

  Daily IG reel: produce and render

• dailycreator

  How do I grow my Instagram

  Daily IG reel research bundle