Anima

SKILL.md

---
description: Anima Avatar - Interactive Video Generation Engine. Generates 16:9 videos with dynamic character sprites (Shutiao), synced audio (Fish Audio), and text overlay.
---

# Anima Avatar (Project Anima)

Generates high-quality interactive videos where Shutiao speaks the text with appropriate expressions, gestures, and voice.

## Capabilities
- **True Voice**: Uses Fish Audio API for realistic speech synthesis.
- **Dynamic Sprites**: Auto-selects from a library of 30+ sprites (Happy, Angry, Shy, Think, Action) based on emotion tags.
- **Smart Director**: Handles parallel rendering, audio-sync, and video composition (FFmpeg).
- **Pro Delivery**: Uploads as native stream to Feishu for direct playback (with correct duration).

## Structure
- `src/director.js`: The core engine. Generates frames (sharp + SVG), audio (Fish Audio), and video (FFmpeg).
- `src/send_video_pro.js`: Delivery script. Handles transcoding, duration calculation, and Feishu upload.
- `src/batch_generator.js`: Batch sprite generator. Uses Gemini image generation to produce sprite variants.
- `assets/sprites/`: The sprite library (1920x1080 PNG files).
- `assets/production_plan.csv`: The asset registry (25 sprites).
- `assets/manifest.json`: Sprite metadata for reference.
- `output/`: Generated videos.

## IMPORTANT: Sprites Not Included

**ClawHub only distributes text files.** The sprite PNG images are **not included** in the published package.

After installing, follow the steps below **in order** to prepare your sprites before first use.

All image generation steps use **Gemini API (Nano Banana)** as the AI image generator. It works by "reference image + text prompt" — you give it an existing image and a text description of what to change, and it returns a new image with the changes applied. This is how both the base sprite (character + background fusion) and all expression variants are created.

### Step 1: Prepare your character image

You need a **standalone character illustration** (transparent background PNG recommended).

- This is your character's "identity" — it defines the look for all sprites.
- Resolution: at least 1920x1080. Full-body is best.
- Example: a full-body anime character PNG with transparent background.

Save it somewhere accessible (e.g. `avatars/my_character.png`).

### Step 2: Prepare your background image

You need a **background scene** for the character to stand in.

- This is the environment that appears behind the character in every video frame.
- Resolution: at least 1920x1080.
- Example: a cherry blossom garden, a classroom, a city street.

Save it at: `assets/backgrounds/` (e.g. `assets/backgrounds/cherry_blossom_bg.png`).

### Step 3: Fuse character + background into base sprite

This step uses **Gemini (Nano Banana) image generation** to merge your character onto the background. The AI sees both images and creates a natural-looking composite — this is NOT a simple overlay/paste, but an AI-generated fusion that handles lighting, shadows, and blending.

How to do it:

**Method A: Use Gemini directly (recommended)**
Use any Gemini-compatible image generation tool (like Nano Banana, Google AI Studio, or the Gemini API) with:
- **Input image**: Your background image
- **Reference/overlay**: Your character image
- **Prompt**: e.g. "Place this character naturally in the center of this background scene, full body visible, gentle smile"

Save the output as: `assets/sprites/shutiao_base.png`

**Method B: Use the built-in compose script (simple overlay)**
If you just want a quick mechanical overlay (no AI blending), `src/compose_base.js` can paste your character onto the background using sharp:
1. Edit `src/compose_base.js` — update `BG_PATH` and `AVATAR_PATH` to point to your files.
2. Run: `node src/compose_base.js`
3. Output: `assets/sprites/shutiao_base.png`

Note: Method B is a plain image composite. Method A (Gemini) produces much better results because it handles lighting and integration naturally.

### Step 4: Plan your sprite variants

Now that you have a base sprite, plan what expression/pose variants you want.

Open `assets/production_plan.csv` and customize it:

```
ID,Emotion,Variant,Description,Filename,Prompt,Status
001,Base,v1,Standard,shutiao_base.png,gentle smile looking at viewer,Done
003,Happy,v1,Smile,shutiao_happy.png,big happy smile eyes closed,Pending
007,Angry,v1,Pout,shutiao_angry.png,angry face pouting,Pending
...
```

Column meanings:
- **Emotion**: Category used by the video director to pick sprites (Happy, Angry, Shy, Think, Sad, Action, Base).
- **Filename**: Output filename. Must follow `shutiao_<emotion>_<variant>.png` format.
- **Prompt**: Describes how this variant differs from the base. The generator sends the base image + this prompt to Gemini, asking it to change only the expression/pose while keeping everything else the same.
- **Status**: `Pending` = will be generated. `Done` = already exists, skip.

The default CSV has 25 entries. You can add, remove, or modify rows freely.

### Step 5: Generate sprite variants

This step uses **Gemini (Nano Banana) image generation** again. For each `Pending` row, the batch generator sends your base sprite + the prompt to Gemini, asking: "Same image, change facial expression to [prompt]. Keep clothes and background exactly same."

1. Set your Gemini API key in `skills/anima/.env`:
```ini
GEMINI_API_KEY=your_key_here
```

2. Make sure `assets/sprites/shutiao_base.png` (or `shutiao_base_1k.png`) exists from Step 3.

3. Run the batch generator:
```bash
node skills/anima/src/batch_generator.js
```

What happens:
- Reads `production_plan.csv`
- Finds all rows with `Status=Pending`
- For each: sends the base sprite + prompt to Gemini API
- Saves the generated image as a PNG in `assets/sprites/`
- Updates the CSV row to `Status=Done`
- Waits 10 seconds between generations (API rate limit cooldown)

### Step 6: Verify

Check that `assets/sprites/` now has a PNG file for every row in `production_plan.csv`:
```bash
ls assets/sprites/*.png | wc -l
```

Then do a quick test run:
```bash
node skills/anima/run.js --preview --script '[{"text":"Test","emotion":"Happy"}]'
```

Check the generated frame at `temp/frame_0.png` — you should see your character with the text overlay.

If a sprite is missing at runtime, the director will fall back to a white background with a warning in the console.

## Setup & Requirements

### 1. System Dependencies
- **ffmpeg** (required for video processing):
  - macOS: `brew install ffmpeg`
  - Linux: `sudo apt install ffmpeg`
  - Windows: Download/Install FFmpeg and add to PATH.

### 2. Node Dependencies
Install inside the skill folder:
```bash
cd skills/anima
npm install
```

The only native dependency is `sharp`, which ships prebuilt binaries for all major platforms via N-API. It does **not** need recompilation when Node versions change — install once, run everywhere.

### 3. External Services (API Keys Required)

This skill depends on two external services. You need to provide your own API keys.

#### Fish Audio (TTS - Text to Speech)
- **What**: Generates realistic voice audio from text.
- **Used by**: `src/director.js` (the `generateAudio()` function).
- **Get a key**: https://fish.audio/dashboard/api
- **Env vars needed**:
  - `FISH_AUDIO_KEY` — Your API key (starts with `sk-...` or a hex string).
  - `FISH_AUDIO_REF_ID` — The voice model reference ID. You can use Fish Audio's default models or clone your own voice.

#### Gemini API (Image Generation - Optional)
- **What**: Generates sprite variants using Google Gemini image generation.
- **Used by**: `src/batch_generator.js` (only needed if you want to create new sprite variants).
- **Self-contained**: No external skills needed. `batch_generator.js` calls the Gemini API directly via curl.
- **Get a key**: https://aistudio.google.com/apikey
- **Env var needed**: `GEMINI_API_KEY`
- **Not needed** for normal video generation — only for creating new character sprites.

#### Feishu / Lark (Delivery - Optional)
- **What**: Uploads videos to Feishu as native media messages.
- **Used by**: `src/send_video_pro.js`.
- **Env vars needed**:
  - `FEISHU_APP_ID` — Your Feishu app ID.
  - `FEISHU_APP_SECRET` — Your Feishu app secret.
- **Not needed** if you only use `--preview` mode.

### 4. Environment Configuration
Create a `.env` file **inside the skill folder** (`skills/anima/.env`):
```ini
# Fish Audio (Required for TTS)
FISH_AUDIO_KEY=your_key_here
FISH_AUDIO_REF_ID=your_model_ref_id_here

# Gemini (Optional, for sprite generation)
GEMINI_API_KEY=your_key_here

# Feishu/Lark (Optional, for delivery)
FEISHU_APP_ID=cli_...
FEISHU_APP_SECRET=...
```

**Important**: The `.env` file is loaded from the skill folder first (least-privilege). Never commit `.env` files — the `.clawignore` already excludes it.

## Usage

### Generate & Send
```bash
# Basic usage (Demo script)
node skills/anima/run.js --target "ou_..."

# With custom script (JSON string)
node skills/anima/run.js --target "ou_..." --script '[{"text":"Hello World","emotion":"Happy"}]'

# With custom script (File)
node skills/anima/run.js --target "ou_..." --script "path/to/script.json"

# Preview only (No upload)
node skills/anima/run.js --script '[{"text":"Test","emotion":"Happy"}]' --preview
```

### One-Liner (for agent use)
```bash
node skills/anima/run.js --target "<open_id>" --script '[{"text":"Hello","emotion":"Happy"}]'
```

## Script Format
Each scene in the script is a JSON object:
```json
[
  { "text": "Hello boss!", "emotion": "Happy" },
  { "text": "Let me think...", "emotion": "Think" },
  { "text": "I got it!", "emotion": "Action" }
]
```

**Available emotions**: `Base`, `Happy`, `Angry`, `Shy`, `Think`, `Sad`, `Action`.

## Extension: Custom TTS

To use a different TTS provider (e.g., OpenAI, ElevenLabs):

1. Open `src/director.js`.
2. Locate the `generateAudio(text, filename)` function.
3. Replace the Fish Audio API call with your provider's logic.
4. **Contract**: The function must return: `{ path: "/path/to/audio.wav", duration: 1.5 }` (duration in seconds).

## Advanced: Adding More Sprite Variants

To add new expressions or poses after the initial setup:

1. Add a new row to `assets/production_plan.csv` with `Status=Pending`.
2. Write a clear prompt describing the change from the base (e.g. `angry expression, arms crossed, looking away`).
3. Run `node src/batch_generator.js` — it will only process `Pending` rows.
4. The new sprite will auto-register in the director's emotion pool via `loadSprites()`.

See `ASSETS_PLAN.md` for the full production matrix and design philosophy.

## Troubleshooting
- **Duration 00:00**: Ensure `send_video_pro.js` calculates duration in **ms** and passes it to both upload and message payload.
- **Fish Audio 400**: Check that your Ref ID matches the API Key owner's model.
- **Video Black**: Check `ffmpeg` transcoding logs and verify source frame images in `temp/frame_*.png`.
- **SVG text not rendering**: Ensure the system has CJK fonts installed (macOS has them by default; on Linux: `sudo apt install fonts-noto-cjk`).
- **No audio fallback**: If `FISH_AUDIO_KEY` is missing, the skill falls back to macOS `say` command (English only).