n8n Pilot

SKILL.md

---
name: n8n-pilot
version: 0.1.0
description: "Design, build, deploy, test, and secure advanced n8n workflows. Architecture patterns, flow logic, dangerous pattern detection, self-hosting, credential management, and workflow recipes. Not an API wrapper — a workflow architect."
changelog: "v0.1.0 MVP: design patterns, flow logic, self-hosting, credentials, testing, security, dangerous pattern detection, 3 core recipes"
metadata: {"clawdbot":{"emoji":"⚙️","requires":{"env":["N8N_API_KEY","N8N_BASE_URL"]},"primaryEnv":"N8N_API_KEY"}}
---

# n8n Pilot ⚙️

Design, build, deploy, test, and secure advanced n8n workflows. Not an API wrapper — a workflow architect that guides you from intent to production.

## When to Use

Use when the task involves n8n — designing workflows, building automations, deploying instances, diagnosing failures, optimizing performance, or securing webhooks/credentials.

## Companion Skills

- `clawhub install n8n` — API client for CRUD operations (list/create/activate workflows)
- `clawhub install docker-pilot` — Container lifecycle management for deploying n8n

---

## Safety Architecture ⚠️

### Dangerous Pattern Detection

Before deploying ANY workflow, scan for these patterns:

| Pattern | Risk | Detection |
|---------|------|-----------|
| Loop without max iteration | 🔴 Infinite execution | Split In Batches without `batchSize`, or Code node with `while(true)` |
| Delete without filter | 🔴 Data destruction | Database delete/HTTP DELETE without WHERE/filter parameters |
| Webhook without auth | 🟡 Anyone can trigger | Webhook node with `authentication=none` |
| HTTP to private IPs | 🟡 SSRF risk | HTTP Request to `10.x`, `172.16-31.x`, `192.168.x`, `localhost` |
| Email/Send without limit | 🟡 Mass spam | Loop + email/Send node without max cap |
| No error handling | 🟡 Silent failures | Workflow with no Error Trigger or error branches |
| Hardcoded secrets | 🔴 Credential exposure | Code node containing API keys, passwords, tokens in plaintext |

**Rule:** Workflows with 🔴 patterns MUST be fixed before deployment. 🟡 patterns require user acknowledgment.

### Confirmation Gates

- **New workflow deployment:** Show logic map → confirm → deploy inactive → test → activate
- **Workflow modification:** Show diff (what changed) → confirm → deploy
- **Destructive operations** (delete workflow, prune executions): Full audit + explicit confirmation

### Kill Switch

If a workflow goes haywire:
```bash
# Deactivate all workflows immediately
python3 scripts/n8n_api.py list-workflows --active true | \
  python3 -c "import sys,json; [print(w['id']) for w in json.load(sys.stdin)['data']]" | \
  xargs -I{} python3 scripts/n8n_api.py deactivate --id {}
```

---

## Workflow Design: Intent → Logic Map → JSON

### Step 1: Understand Intent

Before building, articulate the workflow in plain language:

```
"I want to be notified on Telegram when an important email arrives from my boss."
```

### Step 2: Create Logic Map

Translate intent into a logic map (show to user for confirmation):

```
📧 Email Trigger (new email)
  → 🧠 IF node (sender contains "boss@company.com")
    → ✅ Yes: Summarize email content
      → 📱 Telegram notification (summary)
    → ❌ No: Skip
```

### Step 3: Generate Workflow JSON

Convert the logic map into n8n workflow JSON structure:

```json
{
  "name": "Boss Email → Telegram Alert",
  "nodes": [...],
  "connections": {...},
  "settings": {
    "executionOrder": "v1",
    "timezone": "Europe/Berlin"
  }
}
```

**Key JSON fields:**
- `typeVersion` — must match the node version installed in your n8n instance
- `position` — [x, y] coordinates for UI rendering (required by API)
- `credentials` — embedded as `{credential_type: {id, name}}`, not just an ID
- `parameters` — node-specific config, use `=` prefix for expressions
- `settings.executionOrder` — must be `"v1"` for modern n8n
- `pinData` — inject mock data per-node for testing

---

## Flow Logic Patterns

### Branching (IF / Switch)

```
Trigger → IF (condition)
  → True branch: [process A]
  → False branch: [process B or Stop]
```

**IF Node config:**
```json
{
  "type": "n8n-nodes-base.if",
  "parameters": {
    "conditions": {
      "boolean": [{
        "value1": "={{ $json.sender }}",
        "operation": "contains",
        "value2": "boss@company.com"
      }]
    }
  }
}
```

**Switch Node** (multi-way branching):
```json
{
  "type": "n8n-nodes-base.switch",
  "parameters": {
    "rules": [
      { "output": 0, "conditions": {...} },
      { "output": 1, "conditions": {...} }
    ]
  }
}
```

### Merging (Merge Node)

```
Branch A → Merge → Continue
Branch B ↗
```

**Modes:**
- `append` — combine all items from all inputs
- `mergeByPosition` — zip items by index
- `mergeByKey` — join on a shared field (like SQL JOIN)

### Looping (Split In Batches / Loop Over Items)

```
Large Dataset → Split In Batches (size=100)
  → Process batch
  → Loop back
→ Continue when done
```

**⚠️ Always set `batchSize`** — prevents memory exhaustion on large datasets.

### Sub-Workflows (Execute Workflow Node)

```
Main Workflow → Execute Workflow (sub-workflow ID)
  → Use sub-result in main flow
```

**Use cases:** Reusable logic (e.g., "send notification" sub-workflow called by 5 different main workflows), breaking complex flows into manageable pieces.

**Important:** Sub-workflow must have a trigger that accepts calls from parent (`Workflow Trigger` node).

### Error Handling (Error Trigger + Error Branches)

**Pattern 1: Error Trigger Node**
```
[Separate workflow]
Error Trigger → Notify (Telegram/Email) with error details
```

**Pattern 2: Per-node error branch**
```
HTTP Request → [on error] → Log error + fallback action
             → [on success] → Continue normal flow
```

### Retry with Exponential Backoff

n8n has no built-in retry with backoff. Implement in Code node:

```javascript
// In a Code node, wrapping an API call
const maxRetries = 3;
const baseDelay = 1000; // 1 second

for (let attempt = 1; attempt <= maxRetries; attempt++) {
  try {
    const response = await this.helpers.httpRequest({
      url: 'https://api.example.com/data',
      method: 'GET',
    });
    return [{ json: response }];
  } catch (error) {
    if (attempt === maxRetries) throw error;
    const delay = baseDelay * Math.pow(2, attempt - 1);
    await new Promise(resolve => setTimeout(resolve, delay));
  }
}
```

---

## Core Node Catalog

### Trigger Nodes (Start a Workflow)

| Node | Use When | Key Config |
|------|----------|------------|
| Manual Trigger | Testing, one-off runs | No config — click "Execute" |
| Webhook | External systems calling n8n | Path, method (GET/POST), authentication |
| Schedule Trigger | Recurring (cron) tasks | Cron expression or interval |
| Email Trigger (IMAP) | New email arrives | IMAP credentials, folder, filter |
| Polling Trigger | Check API periodically | URL, interval, pagination |

### Flow Control Nodes

| Node | Use When | Key Config |
|------|----------|------------|
| IF | Binary branching | Conditions (boolean/string/number) |
| Switch | Multi-way branching | Rules with output indices |
| Merge | Combining branches | Mode (append/mergeByPosition/mergeByKey) |
| Split In Batches | Process large datasets in chunks | Batch size (always set this!) |
| Loop Over Items | Process items one at a time | Can use Split In Batches instead |
| Wait | Delay execution | Time (seconds) or specific time |
| No Operation | Placeholder / passthrough | Does nothing — useful for routing |

### Action Nodes

| Node | Use When | Key Config |
|------|----------|------------|
| HTTP Request | Any REST/GraphQL API | URL, method, headers, auth, body |
| Code (JavaScript) | Custom logic, transformation | Input items, access via `$input` |
| Set | Add/modify fields | Field assignments, expressions |
| Date & Time | Timezone conversion, formatting | Format string, timezone |
| Crypto | Hash, encrypt, sign | Algorithm, key |
| Spreadsheet File | Read/write Excel/CSV | File format, options |

### App Nodes (Common Integrations)

| Node | For | Auth Type |
|------|-----|-----------|
| Gmail | Send/read emails | OAuth2 |
| Google Sheets | Read/write spreadsheets | OAuth2 |
| Google Calendar | Event management | OAuth2 |
| Telegram | Send messages, bots | Bot token |
| Slack | Messages, channels | OAuth2 or Bot token |
| PostgreSQL | Database queries | Connection string |
| MySQL | Database queries | Connection string |
| MongoDB | Document queries | Connection string |
| Redis | Key-value operations | Connection string |
| GitHub | Repos, issues, PRs | Personal access token or OAuth2 |
| Discord | Messages, webhooks | Bot token |

---

## Self-Hosting Guide

### Development (Single Container)

```yaml
# docker-compose.yml
services:
  n8n:
    image: docker.n8n.io/n8nio/n8n
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - N8N_ENCRYPTION_KEY=<generate-a-random-32-char-key>
      - WEBHOOK_URL=http://localhost:5678
      - TZ=Europe/Berlin
    volumes:
      - n8n_data:/home/node/.n8n
      - /etc/localtime:/etc/localtime:ro

volumes:
  n8n_data:
```

### Production (Queue Mode — PostgreSQL + Redis)

```yaml
# docker-compose.yml — production
services:
  postgres:
    image: postgres:16-alpine
    restart: unless-stopped
    environment:
      POSTGRES_USER: n8n
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_DB: n8n
    volumes:
      - pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U n8n"]
      interval: 30s
      timeout: 5s
      retries: 3

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    command: redis-server --appendonly yes --maxmemory 256mb --maxmemory-policy allkeys-lru
    volumes:
      - redis_data:/data
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]
      interval: 30s
      timeout: 5s
      retries: 3

  n8n-main:
    image: docker.n8n.io/n8nio/n8n
    container_name: n8n
    restart: unless-stopped
    ports:
      - "5678:5678"
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=${DB_PASSWORD}
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_ENCRYPTION_KEY=${ENCRYPTION_KEY}
      - WEBHOOK_URL=https://n8n.example.com
      - TZ=Europe/Berlin
      - EXECUTIONS_DATA_PRUNE=true
      - EXECUTIONS_DATA_MAX_AGE=336
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy
    volumes:
      - n8n_data:/home/node/.n8n

  n8n-worker:
    image: docker.n8n.io/n8nio/n8n
    restart: unless-stopped
    command: worker --concurrency=5
    environment:
      - DB_TYPE=postgresdb
      - DB_POSTGRESDB_HOST=postgres
      - DB_POSTGRESDB_DATABASE=n8n
      - DB_POSTGRESDB_USER=n8n
      - DB_POSTGRESDB_PASSWORD=${DB_PASSWORD}
      - EXECUTIONS_MODE=queue
      - QUEUE_BULL_REDIS_HOST=redis
      - N8N_ENCRYPTION_KEY=${ENCRYPTION_KEY}
    depends_on:
      postgres:
        condition: service_healthy
      redis:
        condition: service_healthy

volumes:
  pgdata:
  redis_data:
  n8n_data:
```

### Critical Environment Variables

| Variable | Purpose | Default | Production |
|----------|---------|---------|------------|
| `N8N_ENCRYPTION_KEY` | AES-256 key for credentials | Auto-generated | **Must be set and shared across all nodes** |
| `EXECUTIONS_MODE` | `regular` or `queue` | `regular` | `queue` for production |
| `DB_TYPE` | Database backend | SQLite | `postgresdb` for queue mode |
| `WEBHOOK_URL` | Public URL for webhooks | `http://localhost:5678` | `https://n8n.example.com` |
| `EXECUTIONS_DATA_PRUNE` | Auto-delete old executions | `false` | `true` |
| `EXECUTIONS_DATA_MAX_AGE` | Retention in hours | 336 (14d) | Set per policy |
| `N8N_GRACEFUL_SHUTDOWN_TIMEOUT` | Worker drain time (seconds) | 30 | 60+ for long workflows |
| `QUEUE_HEALTH_CHECK_ACTIVE` | Worker health endpoints | `false` | `true` |

### ⚠️ Queue Mode Requirements

- **PostgreSQL required** — SQLite does NOT work in queue mode
- **N8N_ENCRYPTION_KEY must be identical** across main + all workers
- **Community nodes must be installed on ALL containers** (main + every worker)

### Backup Strategy

```bash
# What to back up:
# 1. PostgreSQL database (pg_dump)
docker exec postgres pg_dump -U n8n n8n > n8n_backup_$(date +%Y%m%d).sql

# 2. n8n data volume (encryption key, config)
docker cp n8n:/home/node/.n8n ./n8n_data_backup/

# 3. .env file with passwords and encryption key
```

**Critical:** If you lose `N8N_ENCRYPTION_KEY`, ALL credentials become permanently unrecoverable. Back it up securely.

---

## Credential Management

### Creating Credentials

Credentials can be created via the REST API (`POST /credentials`) but **OAuth2 credentials require browser interaction** for the consent flow.

```bash
# Create an API key credential via API
curl -X POST "${N8N_BASE_URL}/api/v1/credentials" \
  -H "X-N8N-API-KEY: ${N8N_API_KEY}" \
  -H "Content-Type: application/json" \
  -d '{
    "name": "My API Key",
    "type": "httpHeaderAuth",
    "data": {
      "name": "Authorization",
      "value": "Bearer sk-xxx"
    }
  }'
```

### OAuth2 Credentials

OAuth2 requires a browser redirect. **Cannot be fully automated.** Steps:
1. Create credential via API with `clientId` and `clientSecret`
2. User completes OAuth consent flow in browser
3. n8n stores the refresh/access tokens

### Encryption Key Management

- **CRITICAL:** `N8N_ENCRYPTION_KEY` encrypts ALL credentials with AES-256-GCM
- If lost, credentials are permanently unrecoverable — no reset possible
- Must be identical across main + all workers in queue mode
- Store in a password manager or `.env` file (never in docker-compose.yml in production)
- Rotating the key requires re-creating all credentials

### Credential Types

| Type | Auth Method | Automatable? |
|------|------------|-------------|
| `httpHeaderAuth` | Header-based (Bearer token) | ✅ Yes |
| `httpBasicAuth` | Username + password | ✅ Yes |
| `oAuth2Api` | OAuth2 flow | ❌ Requires browser |
| `telegramBotApi` | Bot token | ✅ Yes |
| `postgresApi` | Connection string | ✅ Yes |
| `mysqlApi` | Connection string | ✅ Yes |
| `smtpAccount` | SMTP credentials | ✅ Yes |
| `slackApi` | Bot token | ✅ Yes |

---

## Testing Strategy

### Pre-Deployment Checklist

1. **Validate structure** — all nodes have name, type, valid connections
2. **Check for dangerous patterns** (see Safety Architecture)
3. **Set `pinData`** — mock input data for testing without real triggers
4. **Deploy inactive** — never activate a new workflow without testing
5. **Manual trigger test** — run with test data, verify output
6. **Error branch test** — intentionally trigger error paths
7. **Activate** — only after all tests pass

### Mock Data with pinData

```json
{
  "nodes": [{
    "name": "Webhook",
    "type": "n8n-nodes-base.webhook",
    "parameters": { "path": "test-webhook" }
  }],
  "pinData": {
    "Webhook": [{ "json": { "email": "test@example.com", "subject": "Test" } }]
  }
}
```

`pinData` lets you test downstream nodes without the trigger actually firing.

### Validation Command

```bash
python3 scripts/n8n_tester.py validate --file workflow.json --pretty
```

### Test Execution

```bash
# Execute with test data
python3 scripts/n8n_api.py execute --id <workflow-id> --data '{"key": "value"}'

# Check result
python3 scripts/n8n_api.py get-execution --id <execution-id> --pretty
```

**⚠️ Note:** n8n has no native dry-run. `execute_workflow` runs the workflow for real. Always deploy workflows in inactive state and test with manual triggers before activating.

---

## Webhook Patterns

### Incoming Webhook (External → n8n)

```
External Service → POST https://n8n.example.com/webhook/my-path → n8n Workflow
```

**Security options:**
- `none` — no auth (⚠️ anyone can trigger)
- `headerAuth` — require specific header
- `basicAuth` — username + password
- `jwtAuth` — JWT token verification

**Best practice:** Always use at least `headerAuth`:

```json
{
  "type": "n8n-nodes-base.webhook",
  "parameters": {
    "path": "my-secure-webhook",
    "authentication": "headerAuth",
    "headerAuth": {
      "name": "X-Webhook-Secret",
      "value": "={{ $env.WEBHOOK_SECRET }}"
    }
  }
}
```

### Responding to Webhooks

By default, n8n responds immediately with `{"message": "Workflow was started"}`. To send custom responses:

```json
{
  "type": "n8n-nodes-base.respondToWebhook",
  "parameters": {
    "respondWith": "json",
    "responseBody": "={{ JSON.stringify({ status: 'ok', id: $json.id }) }}"
  }
}
```

### Webhook URL Construction

- **Production:** `{WEBHOOK_URL}/webhook/{path}`
- **Test:** `{WEBHOOK_URL}/webhook-test/{path}`
- Test webhooks only work when the workflow editor is open in the browser

---

## Workflow Recipes

### Recipe 1: Email → AI Summarize → Telegram

**"The Executive Assistant"**

```
📧 Email Trigger (IMAP)
  → 🧠 Code Node (extract sender, subject, body)
  → 🤖 OpenAI Node (summarize: "Summarize this email in 2 sentences")
  → 📱 Telegram Node (send summary)
  → 🏷️ Set Node (mark email as read)
```

**Logic map:**
"When a new email arrives, extract the content, use AI to summarize it, send the summary to my Telegram, and mark the email as read."

### Recipe 2: Webhook Receiver → Process → Respond

**"The API Processor"**

```
🌐 Webhook (POST /process)
  → 🧪 Validate input (Code node: check required fields)
  → ↔️ IF node (valid input?)
    → ✅ Yes: Process data (HTTP Request to external API)
      → 📤 Respond to Webhook (success + result)
    → ❌ No: Respond to Webhook (400 + error message)
  → 🚨 Error Trigger → Log + notify
```

**Logic map:**
"Receive a webhook, validate the input, process it if valid (or return an error), and respond to the caller."

### Recipe 3: Scheduled Data Sync

**"The Data Synchronizer"**

```
⏰ Schedule Trigger (every 6 hours)
  → 📥 HTTP Request (fetch from source API)
  → 🔀 Split In Batches (batch size: 100)
    → 🗃️ PostgreSQL Node (upsert records)
  → 🧮 Code Node (count synced records)
  → 📱 Telegram Node (sync summary: "Synced 847 records")
  → 🚨 Error Trigger → Log + retry notification
```

**Logic map:**
"Every 6 hours, fetch data from the source API, batch-upsert into the database, and send me a summary on Telegram."

---

## n8n API Gaps (What Requires UI)

| Task | API? | Workaround |
|------|------|------------|
| Create API key credentials | ✅ | `POST /credentials` |
| Install community nodes | ❌ | `npm install` in container + restart |
| OAuth2 consent flow | ❌ | Browser required — no automation |
| User management | ❌ | Internal API `/rest/users` or direct DB |
| Project CRUD | ❌ | UI only |
| Workflow variables | ❌ | Use `staticData` in Code nodes |
| License management | ❌ | UI only |

---

## Integration with OpenClaw Ecosystem

### Docker Pilot

Delegate n8n deployment to Docker Pilot:
- n8n-pilot provides the Compose files
- Docker Pilot handles container lifecycle (start/stop/restart/health checks)
- Docker Pilot protects the n8n container as a critical service

### Council of LLMs

Use n8n webhooks to trigger Council deliberations:
```
n8n Webhook → HTTP Request to OpenClaw → Spawn Council → Return result → n8n continues
```

### Thrift-Cycle

n8n as the execution engine for periodic eBay data collection:
```
Schedule Trigger → Run Thrift-Cycle pipeline → Parse results → Alert on hot items
```

---

## Quick Reference

| Task | Command / Pattern |
|------|-------------------|
| List workflows | `python3 scripts/n8n_api.py list-workflows --pretty` |
| Get workflow | `python3 scripts/n8n_api.py get-workflow --id ID --pretty` |
| Create workflow | `python3 scripts/n8n_api.py create --from-file workflow.json` |
| Activate | `python3 scripts/n8n_api.py activate --id ID` |
| Deactivate | `python3 scripts/n8n_api.py deactivate --id ID` |
| Execute | `python3 scripts/n8n_api.py execute --id ID --data '{}'` |
| Validate | `python3 scripts/n8n_tester.py validate --file workflow.json --pretty` |
| Check executions | `python3 scripts/n8n_api.py list-executions --limit 10 --pretty` |
| Performance | `python3 scripts/n8n_optimizer.py analyze --id ID --pretty` |
| Create credential | `curl -X POST BASE_URL/api/v1/credentials -H "X-N8N-API-KEY: KEY"` |
| Backup DB | `docker exec postgres pg_dump -U n8n n8n > backup.sql` |
| Install community node | `docker exec n8n npm install n8n-nodes-NAME && docker restart n8n` |

---

## First-Run Setup

When activating n8n-pilot on a new machine:

1. **Detect n8n instance** — check if port 5678 is listening, or scan Docker containers
2. **Verify API access** — test `GET /api/v1/workflows` with API key
3. **Audit existing workflows** — list all, check for failures, run dangerous pattern scan
4. **Configure credentials** — set up essential credentials (Telegram bot, email, database)
5. **Deploy if missing** — use Docker Pilot to deploy n8n with the self-hosting Compose files
6. **Enable monitoring** — set up health checks and execution pruning

---

## Credits

Extends the `n8n` skill by thomasansems (v2.0.0). This skill adds:
- ⚙️ Workflow design patterns (branching, merging, looping, sub-workflows)
- 🛡️ Dangerous pattern detection and safety architecture
- 🐳 Self-hosting guide (dev + production queue mode)
- 🔑 Credential management (creation, OAuth, encryption key)
- 🧪 Testing strategy (pinData, validation, pre-deployment checklist)
- 🔒 Webhook security patterns
- 📋 3 core workflow recipes
- 🚀 First-run setup and integration with Docker Pilot