ComfyUI

SKILL.md

---
name: ComfyUI
description: Run local ComfyUI workflows via the HTTP API. Use when the user asks to run ComfyUI, execute a workflow by file path/name, or supply raw API-format JSON; supports the default workflow bundled in assets.
read_when:
  - User asks to generate images with ComfyUI
  - User provides a workflow file or JSON to run
  - User describes an image to generate (subject, style, scene)
  - User pastes or sends a list of model weight URLs to download for ComfyUI
metadata: {"clawdbot":{"emoji":"🖼️","requires":{"bins":["python3"]}}}
---

# ComfyUI Runner

## Overview
Run ComfyUI workflows on the local server (default 127.0.0.1:8188) using API-format JSON and return output images.

## Editing the workflow before running
The run script only takes `--workflow <path>`. You must **inspect and edit the workflow JSON** before running, using your best knowledge of the ComfyUI API format. Do not assume fixed node IDs, `class_type` names, or `_meta.title` values — the user may have updated the default workflow or supplied a custom one.

**For every run (including the default workflow):**
1. Read the workflow JSON (default: `skills/comfyui/assets/default-workflow.json`, or the path/file the user gave).
2. **Identify prompt-related nodes** by inspecting the graph: look for nodes that hold the main text prompt — e.g. `PrimitiveStringMultiline`, `CLIPTextEncode` (positive text), or any node with `_meta.title` or `class_type` suggesting "Prompt" / "positive" / "text". Update the corresponding input (e.g. `inputs.value`, or the text input to the encoder) to the image prompt you derived from the user (subject, style, lighting, quality). If the user didn’t ask for a custom image, you can leave the existing prompt or tweak only if needed.
3. **Optionally identify style/prefix nodes** — e.g. `StringConcatenate`, or a second string input that acts as style. Set them if the user asked for a specific style or to clear a default prefix.
4. **Optionally set a new seed** — find sampler-like nodes (e.g. `KSampler`, `BasicGuider`, or any node with a `seed` input) and set `seed` to a new random integer so each run can differ.
5. Write the modified workflow to a temp file (e.g. `skills/comfyui/assets/tmp-workflow.json`). Use `~/ComfyUI/venv/bin/python` for any inline Python; do not use bare `python`.
6. Run: `comfyui_run.py --workflow <path-to-edited-json>`.

If the workflow structure is unclear or you can’t find prompt/sampler nodes, run the file as-is and only change what you can reliably identify. Same approach for arbitrary user-supplied JSON: inspect first, edit at your best knowledge, then run.

## Run script (single responsibility)
```bash
~/ComfyUI/venv/bin/python skills/comfyui/scripts/comfyui_run.py \
  --workflow <path-to-workflow.json>
```

The script only queues the workflow and polls until done. It prints JSON with `prompt_id` and output `images`. All prompt/style/seed changes are done by you in the JSON beforehand.

## If the server isn’t reachable
If the run script fails with a connection error (e.g. connection refused or timeout to 127.0.0.1:8188), ComfyUI may not be installed or not running.

**Check:** Does `~/ComfyUI` exist and contain `main.py`?

- **If not installed:** Install ComfyUI (e.g. clone the repo, create a venv, install dependencies, then start the server). Example:
  ```bash
  git clone https://github.com/comfyanonymous/ComfyUI.git ~/ComfyUI
  cd ~/ComfyUI
  python3 -m venv venv
  ~/ComfyUI/venv/bin/pip install -r requirements.txt
  ```
  Then start the server (see below). Tell the user they may need to install model weights into `~/ComfyUI/models/` depending on the workflow.

- **If installed but not running:** Start the ComfyUI server so the API is available on port 8188. Example:
  ```bash
  ~/ComfyUI/venv/bin/python ~/ComfyUI/main.py --listen 127.0.0.1
  ```
  Run in the background or in a separate terminal so it keeps running. Then retry the workflow run.

Use `~` (or the user’s home) for paths so it works on their machine.

## Model weights from URLs
When the user pastes or sends a **list of model weight URLs** (one per line, or comma-separated), download those files into the ComfyUI installation so the workflow can use them later.

1. **Normalize the list** — one URL per line; strip empty lines and comments (lines starting with `#`).
2. **Run the download script** with the ComfyUI base path (default `~/ComfyUI`). The script uses [pget](https://github.com/replicate/pget) for parallel downloads when available; if `pget` is not in PATH, it installs it to `~/.local/bin` automatically (no sudo). If pget cannot be installed (e.g. unsupported OS/arch), it falls back to a built-in download. Use the ComfyUI venv Python so the script runs correctly:
   ```bash
   ~/ComfyUI/venv/bin/python skills/comfyui/scripts/download_weights.py --base ~/ComfyUI
   ```
   Pass URLs as arguments, or pipe a file/list on stdin:
   ```bash
   echo "https://example.com/model.safetensors" | ~/ComfyUI/venv/bin/python skills/comfyui/scripts/download_weights.py --base ~/ComfyUI
   ```
   Or save the user’s list to a temp file and run:
   ```bash
   ~/ComfyUI/venv/bin/python skills/comfyui/scripts/download_weights.py --base ~/ComfyUI < /tmp/weight_urls.txt
   ```
   To force the built-in download (no pget): add `--no-pget`.
3. **Subfolder:** The script infers the ComfyUI models subfolder from the URL/filename (e.g. `vae`, `clip`, `loras`, `checkpoints`, `text_encoders`, `controlnet`, `upscale_models`). The user can optionally specify a subfolder per line as `url subfolder` (e.g. `https://.../model.safetensors vae`). You can also pass a default with `--subfolder loras` so all URLs in that run go to `models/loras/`.
4. **Existing files:** By default the script skips URLs that already exist on disk; use `--overwrite` to replace.
5. **Paths:** Files are written under `~/ComfyUI/models/<subfolder>/`. Tell the user where each file was saved and that they can run the workflow once the ComfyUI server is (re)started if needed.

Supported subfolders (under `ComfyUI/models/`): `checkpoints`, `clip`, `clip_vision`, `controlnet`, `diffusion_models`, `embeddings`, `loras`, `text_encoders`, `unet`, `vae`, `vae_approx`, `upscale_models`, and others. Use `--subfolder <name>` when the auto-inference is wrong.

## After run
Outputs are saved under `ComfyUI/output/`. Use the `images` list from the script output to locate the files (filename + subfolder).

### ⚠️ Always send the output to the user
After a successful ComfyUI run, **you must deliver the generated image(s) to the user**. Do not reply with only the filename in text or with NO_REPLY.

1. Parse the script output JSON for `images` (each has `filename`, `subfolder`, `type`).
2. Build the full path: `ComfyUI/output/` + subfolder + filename (e.g. `ComfyUI/output/z-image_00007_.png`).
3. **Send the image to the user** via the channel they're on (e.g. use the message/send tool with the image `path` so the user receives the file). Include a short caption if helpful (e.g. "Here you go." or "Tokyo street scene.").

Every successful run must result in the user receiving the image. Never leave them with only a filename or no delivery.

## Resources

### scripts/
- `comfyui_run.py`: Queue a workflow, poll until completion, print `prompt_id` and `images`. No args — you edit the JSON before running.
- `download_weights.py`: Download model weight URLs into `~/ComfyUI/models/<subfolder>/`. Uses [pget](https://github.com/replicate/pget) when available (installs to `~/.local/bin` if missing); fallback to built-in download. Input: URLs as args or one per line on stdin. Options: `--base`, `--subfolder`, `--overwrite`, `--no-pget`. Infers subfolder from URL/filename when not given.

### assets/
- `default-workflow.json`: Default workflow. Copy and edit (prompt, style, seed) then run with the edited path; or run as-is for a generic run.