Pve Automation

SKILL.md

---
name: pve-automation
description: Automate Proxmox VE (PVE) virtual machine and container management tasks including VM lifecycle operations (create, start, stop, delete), template management, cloud-init configuration, plugin installation, and cluster operations. Use this skill whenever the user mentions Proxmox, PVE, VM management, container (LXC) operations, qemu, or virtualization automation.
---

# Proxmox VE Automation Skill

This skill provides comprehensive automation for Proxmox VE (Virtual Environment) management through its REST API.

## Overview

Proxmox VE is an open-source server virtualization management platform that supports:
- **QEMU/KVM Virtual Machines** (VMs)
- **LXC Containers** (Linux Containers)
- **Storage management**
- **Cluster management**
- **Backup and snapshots**

## API Basics

- **Base URL**: `https://<pve-host>:<port>/api2/json/`
  - Default port: `8006`
  - Custom port can be configured in PVE settings
- **Authentication**: Ticket-based (cookie) or API Token-based
- **Data format**: JSON
- **API Explorer**: `https://<pve-host>:<port>/pve-docs/api-viewer/`

### Custom API Port

PVE API port can be customized from the default 8006:

```bash
# Check current API port configuration
grep "port" /etc/default/pveproxy

# Or via API: GET /nodes/{node}/services
```

**Using custom port in code:**
```python
# Option 1: Include port in host
client = PVEClient(host='pve.example.com:8443', ...)

# Option 2: Explicit port parameter
class PVEClient:
    def __init__(self, host, port=8006, ...):
        self.base_url = f"https://{host}:{port}/api2/json"

# Option 3: Environment variable
port = os.environ.get('PVE_PORT', 8006)
```

## Authentication Methods

### Method 1: API Token (Recommended for automation)

API tokens provide stateless access without CSRF tokens:

```bash
# Header format
Authorization: PVEAPIToken=USER@REALM!TOKENID=UUID

# Example
curl -H 'Authorization: PVEAPIToken=root@pam!automation=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee' \
  https://pve.example.com:8006/api2/json/nodes
```

**Creating an API Token:**
1. Web UI: Datacenter → Permissions → API Tokens → Add
2. Or via API: `POST /access/users/{userid}/token/{tokenid}`

### Method 2: Ticket-based Authentication

For interactive sessions or when API tokens aren't available:

```bash
# 1. Get ticket and CSRF token
curl -k -d 'username=root@pam' --data-urlencode 'password=secret' \
  https://pve.example.com:8006/api2/json/access/ticket

# Response: { "data": { "ticket": "...", "CSRFPreventionToken": "..." } }

# 2. Use ticket in subsequent requests
curl -k -b "PVEAuthCookie=<ticket>" \
  -H "CSRFPreventionToken: <csrf_token>" \
  https://pve.example.com:8006/api2/json/nodes
```

Tickets expire after 2 hours. Renew by passing the old ticket as password.

## Common Operations

### Node Operations

```bash
# List all nodes
GET /nodes

# Get node status
GET /nodes/{node}/status

# Get node resources (CPU, memory, disk usage)
GET /nodes/{node}/resources
```

### VM (QEMU) Operations

```bash
# List VMs on a node
GET /nodes/{node}/qemu

# Get VM status
GET /nodes/{node}/qemu/{vmid}/status/current

# Start VM
POST /nodes/{node}/qemu/{vmid}/status/start

# Stop VM (graceful shutdown)
POST /nodes/{node}/qemu/{vmid}/status/shutdown

# Stop VM (force/poweroff)
POST /nodes/{node}/qemu/{vmid}/status/stop

# Reset VM
POST /nodes/{node}/qemu/{vmid}/status/reset

# Delete VM
DELETE /nodes/{node}/qemu/{vmid}
```

### LXC Container Operations

```bash
# List containers
GET /nodes/{node}/lxc

# Get container status
GET /nodes/{node}/lxc/{vmid}/status/current

# Start container
POST /nodes/{node}/lxc/{vmid}/status/start

# Stop container
POST /nodes/{node}/lxc/{vmid}/status/stop

# Delete container
DELETE /nodes/{node}/lxc/{vmid}
```

### VM/Container Creation

**Create a VM from template:**
```bash
POST /nodes/{node}/qemu

Parameters:
- vmid: Unique VM ID (100-999999999)
- name: VM name
- memory: RAM in MB
- cores: Number of CPU cores
- sockets: Number of CPU sockets
- cpu: CPU type (host, kvm64, etc.)
- net0: Network config (e.g., "virtio,bridge=vmbr0")
- ide2: CD/DVD drive or cloud-init (e.g., "local:iso/ubuntu-22.04.iso,media=cdrom")
- scsi0: Disk (e.g., "local-lvm:32,format=raw")
- ostype: OS type (l26 for Linux 2.6+, win11 for Windows 11)
```

**Create an LXC container:**
```bash
POST /nodes/{node}/lxc

Parameters:
- vmid: Unique container ID
- hostname: Container hostname
- memory: RAM in MB
- swap: Swap in MB
- cores: CPU cores
- ostemplate: Template path (e.g., "local:vztmpl/debian-12-standard_12.0-1_amd64.tar.zst")
- storage: Storage for rootfs
- rootfs: Root filesystem size (e.g., "local-lvm:8")
- net0: Network config (e.g., "name=eth0,bridge=vmbr0,ip=dhcp")
- password: Root password (or use ssh-public-keys)
```

### Template Management

```bash
# List available templates on storage
GET /nodes/{node}/storage/{storage}/content?content=vztmpl

# Download template from Proxmox VE repository
POST /nodes/{node}/storage/{storage}/download-url

Parameters:
- url: Template URL
- filename: Target filename
- content: "vztmpl" for container templates, "iso" for ISO images

# List container templates available for download
GET /nodes/{node}/aplinfo

# Download from aplinfo (official templates)
POST /nodes/{node}/storage/{storage}/aplinfo
```

### Cloud-Init Configuration

Cloud-init allows automatic VM initialization on first boot:

```bash
# Set cloud-init parameters
POST /nodes/{node}/qemu/{vmid}/config

Parameters:
- cicustom: Custom cloud-init config (optional)
- ciuser: Default user name
- cipassword: Default user password
- citype: Config drive type (configdrive2, nocloud)
- ciupgrade: Run package upgrade on boot (0/1)
- sshkeys: Public SSH keys (URL-encoded)
- ipconfig0: IP config (e.g., "ip=192.168.1.100/24,gw=192.168.1.1")
- ipconfig1: Additional network config
- nameserver: DNS server
- searchdomain: DNS search domain
```

**Important**: Cloud-init requires a cloud-init capable image (Ubuntu Cloud Images, Debian Cloud, etc.) and a configured CD/DVD drive (usually ide2) for the config drive.

### Snapshots and Backups

```bash
# List snapshots
GET /nodes/{node}/qemu/{vmid}/snapshot

# Create snapshot
POST /nodes/{node}/qemu/{vmid}/snapshot
Parameters:
- snapname: Snapshot name
- description: Optional description
- vmstate: Include RAM state (1/0)

# Rollback to snapshot
POST /nodes/{node}/qemu/{vmid}/snapshot/{snapname}/rollback

# Delete snapshot
DELETE /nodes/{node}/qemu/{vmid}/snapshot/{snapname}

# Create backup
POST /nodes/{node}/vzdump
Parameters:
- vmid: VM ID to backup
- storage: Target storage
- mode: snapshot/suspend/stop
- compress: Compression level (0-9)
```

### Storage Operations

```bash
# List storage
GET /storage

# Get storage status
GET /nodes/{node}/storage/{storage}/status

# List storage content
GET /nodes/{node}/storage/{storage}/content

# Upload file to storage (ISO, template)
POST /nodes/{node}/storage/{storage}/upload
Content-Type: multipart/form-data
```

### Network Configuration

```bash
# List network interfaces on node
GET /nodes/{node}/network

# Create network bridge
POST /nodes/{node}/network
Parameters:
- iface: Interface name (e.g., "vmbr1")
- type: "bridge"
- bridge_ports: Physical ports to bridge
- autostart: Start on boot (1/0)
```

### Cluster Operations

```bash
# Get cluster status
GET /cluster/status

# Get cluster resources
GET /cluster/resources

# Get cluster tasks
GET /cluster/tasks

# Get task status
GET /nodes/{node}/tasks/{upid}/status

# Wait for task completion by polling task status
```

## Working with Tasks

Most write operations return a task UPID (Unique Process ID). Poll the task status to track completion:

```bash
# Get task status
GET /nodes/{node}/tasks/{upid}/status

Response includes:
- status: "running", "stopped"
- exitstatus: "OK" on success, error message on failure
- pid: Process ID
```

## Best Practices

### 1. VM ID Management
- VM IDs must be unique across the cluster
- Common convention: 100-999 for VMs, 100000+ for containers (though not required)
- Check available VM IDs: `GET /cluster/resources` and filter by type

### 2. Resource Allocation
- Always specify memory and cores when creating VMs
- Leave overhead for the host (don't allocate 100% of resources)
- Use ballooning for dynamic memory: `balloon: 1024` with `memory: 4096`

### 3. Network Configuration
- Use virtio network drivers for best performance
- Configure bridges properly before creating VMs
- For VLANs: `net0: "virtio,bridge=vmbr0,tag=100"`

### 4. Storage
- Use appropriate storage types:
  - `local`: Directory storage
  - `local-lvm`: LVM thin provisioning
  - `zfs`: ZFS storage
  - `nfs`, `iscsi`, `ceph`: Network storage

### 5. Security
- Use API tokens with limited permissions instead of root
- Enable 2FA for user accounts
- Use firewall rules: `/nodes/{node}/qemu/{vmid}/firewall/rules`

## Common Workflows

### Create a VM from Cloud Image

1. Download cloud image ISO to storage
2. Create VM with cloud-init drive
3. Configure cloud-init parameters
4. Start VM

### Clone a Template VM

```bash
POST /nodes/{node}/qemu/{source-vmid}/clone
Parameters:
- newid: New VM ID
- name: New VM name
- target: Target node (for cluster)
- full: Full clone (1) or linked clone (0)
- storage: Target storage for full clone
```

### Bulk Operations

For multiple VMs, loop through and call API for each:
- Always add delays between operations to avoid overwhelming the API
- Check task completion before proceeding
- Handle errors gracefully

## API Parameters Reference

### VM Creation Parameters (POST /nodes/{node}/qemu)

**Required Parameters:**
- `vmid` (integer): Unique VM ID (100-999999999), must be unique across cluster
- `name` (string): VM name (must be valid DNS name)

**Resource Parameters:**
- `memory` (integer): RAM in MB. Examples: 512, 1024, 2048, 4096, 8192
- `balloon` (integer): Minimum memory for ballooning in MB (optional, enables memory overcommit)
- `cores` (integer): Number of CPU cores per socket (1-128)
- `sockets` (integer): Number of CPU sockets (1-4)
- `cpu` (string): CPU type. Common values: `host`, `kvm64`, `qemu64`, `x86-64-v2`
- `vcpus` (integer): Number of hotplugged vCPUs (defaults to cores*sockets)

**Storage Parameters:**
- `scsi0`, `scsi1`, `virtio0`, `virtio1`, `ide0`, `ide1`, `sata0`, etc.
  - Format: `<storage>:<size>` or `<storage>:<volume-name>`
  - Examples: `local-lvm:32`, `zfs:vm-disk-001`, `local-lvm:64,format=raw`
- `efidisk0` (string): EFI system partition. Example: `local-lvm:1,format=raw,efitype=4m`

**Network Parameters:**
- `net0`, `net1`, etc. (string): Network interface config
  - Format: `<model>,<options>`
  - Models: `virtio` (recommended), `e1000`, `rtl8139`, `vmxnet3`
  - Examples:
    - `virtio,bridge=vmbr0` (basic bridge)
    - `virtio,bridge=vmbr0,tag=100` (VLAN 100)
    - `virtio,bridge=vmbr0,ip=192.168.1.10/24,gw=192.168.1.1` (static IP via cloud-init)

**Boot/Media Parameters:**
- `ostype` (string): OS type for optimizations
  - Linux: `l24` (2.4), `l26` (2.6+), `other`
  - Windows: `wxp`, `w2k`, `w2k3`, `w2k8`, `wvista`, `win7`, `win8`, `win10`, `win11`
- `ide2` (string): CD/DVD drive or cloud-init drive
  - ISO: `local:iso/ubuntu-22.04.iso,media=cdrom`
  - Cloud-init: `local-lvm:cloudinit,media=cdrom`
- `boot` (string): Boot order. Example: `order=ide2;scsi0;net0`
- `bios` (string): `seabios` (default) or `ovmf` (UEFI)

**Cloud-Init Parameters:**
- `ciuser` (string): Default username
- `cipassword` (string): Default password (will be hashed)
- `cicustom` (string): Path to custom cloud-init config
- `citype` (string): `configdrive2` or `nocloud`
- `sshkeys` (string): URL-encoded SSH public keys
- `ipconfig0`, `ipconfig1` (string): Network config
  - DHCP: `ip=dhcp`
  - Static: `ip=192.168.1.10/24,gw=192.168.1.1`
- `nameserver` (string): DNS servers (comma-separated)
- `searchdomain` (string): DNS search domain

### LXC Container Parameters (POST /nodes/{node}/lxc)

**Required Parameters:**
- `vmid` (integer): Unique container ID
- `ostemplate` (string): Template path. Example: `local:vztmpl/debian-12-standard_12.0-1_amd64.tar.zst`

**Resource Parameters:**
- `memory` (integer): RAM in MB
- `swap` (integer): Swap in MB
- `cores` (integer): CPU cores
- `cpulimit` (number): CPU limit (0-128, 0=unlimited)
- `cpuunits` (integer): CPU weight (0-500000, default: 1024)

**Storage Parameters:**
- `rootfs` (string): Root filesystem. Format: `<storage>:<size>`
  - Example: `local-lvm:8`, `local-lvm:16,mountoptions=noatime`
- `mp0`, `mp1`, etc. (string): Mount points
  - Example: `local-lvm:32,mp=/data`, `/host/path:mp=/container/path`

**Network Parameters:**
- `net0`, `net1`, etc. (string): Network config
  - Format: `name=<ifname>,bridge=<br>[,options]`
  - Examples:
    - `name=eth0,bridge=vmbr0,ip=dhcp`
    - `name=eth0,bridge=vmbr0,ip=192.168.1.10/24,gw=192.168.1.1`
    - `name=eth0,bridge=vmbr0,tag=100,ip=dhcp` (VLAN)

**System Parameters:**
- `hostname` (string): Container hostname
- `password` (string): Root password
- `ssh-public-keys` (string): SSH public keys (newline-separated)
- `unprivileged` (boolean): Create unprivileged container (0/1)
- `features` (string): Enabled features. Example: `nesting=1,keyctl=1`
- `onboot` (boolean): Start on boot (0/1)
- `startup` (string): Startup order and delay. Example: `order=1,up=5,down=5`

## Safety and Resource Protection

### 1. Check Available Resources Before Creation

Always verify node resources before creating VMs/containers:

```python
def check_node_resources(client, node, required_memory, required_disk):
    """
    Check if node has enough resources for new VM/container.
    Returns (ok, message) tuple.
    """
    # Get node status
    status = client.get(f'nodes/{node}/status')

    memory = status['memory']
    disk = status['rootfs']  # or check specific storage

    # Memory check (leave 10% overhead for host)
    available_memory = memory['free']
    total_memory = memory['total']
    max_usable = total_memory * 0.9
    current_used = total_memory - available_memory

    if current_used + required_memory > max_usable:
        return False, f"Insufficient memory. Available: {available_memory/1024/1024:.1f}GB, Required: {required_memory/1024:.1f}GB"

    # Disk check
    available_disk = disk['avail']
    if required_disk and required_disk > available_disk:
        return False, f"Insufficient disk space. Available: {available_disk/1024/1024/1024:.1f}GB, Required: {required_disk/1024/1024/1024:.1f}GB"

    return True, "Resources OK"

# Usage before creating VM
ok, msg = check_node_resources(client, 'pve1', memory=4096, disk=32*1024*1024*1024)
if not ok:
    raise Exception(f"Cannot create VM: {msg}")
```

### 2. Get Cluster-Wide Resource Usage

```python
def get_cluster_resources_summary(client):
    """Get aggregated resource usage across all nodes"""
    resources = client.get('cluster/resources')

    summary = {
        'nodes': [],
        'vms': [],
        'containers': [],
        'storage': []
    }

    for r in resources:
        if r['type'] == 'node':
            summary['nodes'].append({
                'name': r['node'],
                'cpu': r.get('cpu', 0),
                'maxcpu': r.get('maxcpu', 0),
                'mem': r.get('mem', 0),
                'maxmem': r.get('maxmem', 0),
                'disk': r.get('disk', 0),
                'maxdisk': r.get('maxdisk', 0),
                'status': r['status']
            })
        elif r['type'] == 'qemu':
            summary['vms'].append(r)
        elif r['type'] == 'lxc':
            summary['containers'].append(r)
        elif r['type'] == 'storage':
            summary['storage'].append(r)

    return summary
```

### 3. Safe VM/Container Creation Helper

```python
class SafePVEClient(PVEClient):
    """PVE client with safety checks"""

    MAX_MEMORY_OVERCOMMIT = 1.2  # Allow 20% overcommit with ballooning
    MAX_SINGLE_VM_MEMORY = 128 * 1024  # 128GB max per VM
    MAX_SINGLE_VM_DISK = 10 * 1024 * 1024 * 1024  # 10TB max per VM

    def safe_create_vm(self, node, **params):
        """Create VM with safety checks"""

        # Validate VMID
        vmid = params.get('vmid')
        if not (100 <= vmid <= 999999999):
            raise ValueError(f"VMID {vmid} out of range (100-999999999)")

        # Check if VMID already exists
        try:
            existing = self.get(f'nodes/{node}/qemu/{vmid}/status/current')
            raise ValueError(f"VM {vmid} already exists on {node}")
        except requests.HTTPError as e:
            if e.response.status_code != 404:
                raise

        # Validate memory
        memory = params.get('memory', 0)
        if memory > self.MAX_SINGLE_VM_MEMORY:
            raise ValueError(f"Memory {memory}MB exceeds maximum {self.MAX_SINGLE_VM_MEMORY}MB")

        # Check node resources
        node_status = self.get(f'nodes/{node}/status')
        total_mem = node_status['memory']['total']
        used_mem = node_status['memory']['used']
        available_mem = total_mem - used_mem

        # Check if all VMs on node can fit (with overcommit)
        all_vms = self.get(f'nodes/{node}/qemu')
        total_allocated = sum(vm.get('maxmem', 0) for vm in all_vms) + memory
        max_allowed = total_mem * self.MAX_MEMORY_OVERCOMMIT

        if total_allocated > max_allowed:
            raise ValueError(
                f"Memory overcommit exceeded. Total allocated would be {total_allocated/1024/1024:.1f}GB, "
                f"max allowed is {max_allowed/1024/1024:.1f}GB"
            )

        # Validate disk size if specified
        for key, value in params.items():
            if key in ['scsi0', 'scsi1', 'virtio0', 'virtio1', 'ide0', 'ide1']:
                # Parse size from disk spec
                if ':' in str(value):
                    try:
                        size_str = str(value).split(':')[1].split(',')[0]
                        if size_str.isdigit():
                            size_gb = int(size_str)
                            if size_gb * 1024 * 1024 * 1024 > self.MAX_SINGLE_VM_DISK:
                                raise ValueError(f"Disk size exceeds maximum")
                    except (IndexError, ValueError):
                        pass

        return self.create_vm(node, **params)

    def safe_create_container(self, node, **params):
        """Create LXC container with safety checks"""

        vmid = params.get('vmid')
        if not (100 <= vmid <= 999999999):
            raise ValueError(f"VMID {vmid} out of range")

        # Check for existing VMID
        try:
            existing = self.get(f'nodes/{node}/lxc/{vmid}/status/current')
            raise ValueError(f"Container {vmid} already exists")
        except requests.HTTPError as e:
            if e.response.status_code != 404:
                raise

        # LXC-specific checks
        memory = params.get('memory', 512)
        if memory > self.MAX_SINGLE_VM_MEMORY:
            raise ValueError(f"Memory {memory}MB exceeds maximum")

        return self.create_container(node, **params)
```

### 4. Dry-Run Mode for Bulk Operations

```python
def bulk_vm_operation(client, node, vmids, operation, dry_run=True):
    """
    Perform bulk operations with dry-run support.
    Operations: 'start', 'stop', 'shutdown', 'delete'
    """
    results = []

    for vmid in vmids:
        try:
            # Verify VM exists and get current status
            vm_status = client.get(f'nodes/{node}/qemu/{vmid}/status/current')
            current_state = vm_status['status']

            if dry_run:
                results.append({
                    'vmid': vmid,
                    'operation': operation,
                    'dry_run': True,
                    'current_state': current_state,
                    'would_execute': True,
                    'message': f"Would {operation} VM {vmid} (currently {current_state})"
                })
            else:
                # Execute operation
                if operation == 'start' and current_state == 'stopped':
                    result = client.post(f'nodes/{node}/qemu/{vmid}/status/start')
                elif operation == 'stop' and current_state == 'running':
                    result = client.post(f'nodes/{node}/qemu/{vmid}/status/stop')
                elif operation == 'shutdown' and current_state == 'running':
                    result = client.post(f'nodes/{node}/qemu/{vmid}/status/shutdown')
                elif operation == 'delete':
                    # Extra safety: only delete if stopped
                    if current_state != 'stopped':
                        raise ValueError(f"VM {vmid} must be stopped before deletion")
                    result = client.delete(f'nodes/{node}/qemu/{vmid}')
                else:
                    result = {'message': f'No action needed (currently {current_state})'}

                results.append({
                    'vmid': vmid,
                    'operation': operation,
                    'success': True,
                    'result': result
                })

        except Exception as e:
            results.append({
                'vmid': vmid,
                'operation': operation,
                'success': False,
                'error': str(e)
            })

    return results
```

### 5. Safe Defaults Checklist

When generating PVE automation code, ensure these safeguards are included:

1. **Pre-flight Checks:**
   - Verify VMID is unique (not already in use)
   - Check node has sufficient resources
   - Validate storage exists and has space

2. **Resource Limits:**
   - Set reasonable defaults (2GB RAM, 2 cores, 32GB disk)
   - Warn if allocating >50% of node resources to single VM
   - Use ballooning for memory overcommit

3. **Operation Safety:**
   - Always use dry-run for bulk operations
   - Confirm destructive operations (delete, rollback)
   - Check VM is stopped before deletion or disk modification

4. **Error Handling:**
   - Handle 409 Conflict (VMID exists)
   - Handle 507 Insufficient Storage
   - Handle 595 PVE errors with meaningful messages

5. **Network Safety:**
   - Validate bridge exists before creating VM
   - Warn about IP conflicts (manual check required)
   - Use VLAN tags appropriately

## Error Handling

Common HTTP status codes:
- `400`: Bad request (missing/invalid parameters)
- `401`: Unauthorized (invalid credentials)
- `403`: Forbidden (insufficient permissions)
- `404`: Resource not found
- `409`: Conflict (VMID already exists)
- `500`: Internal server error
- `507`: Insufficient Storage
- `595`: PVE specific errors (check response body)

Error response format:
```json
{
  "errors": {
    "parameter_name": "error message"
  },
  "message": "Human readable error message"
}
```

### Common Error Patterns and Solutions

| Error | Cause | Solution |
|-------|-------|----------|
| `vmid already exists` | VMID in use | Query `/cluster/resources` to find next available |
| `unable to create VM: out of memory` | Node overloaded | Check `memory.free` before creation |
| `unable to create disk: out of space` | Storage full | Check storage usage before creation |
| `bridge 'vmbrX' does not exist` | Invalid network config | List `/nodes/{node}/network` to get valid bridges |
| `OSTYPE not set` | Missing OS type | Add `ostype` parameter (l26 for Linux, win10 for Windows) |
| `timeout` | Task taking too long | Increase timeout for slow operations (template download) |

## Python Example

```python
import requests
import urllib3
urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)

class PVEClient:
    def __init__(self, host, token_id, token_secret, user='root@pam'):
        self.base_url = f"https://{host}:8006/api2/json"
        self.headers = {
            'Authorization': f'PVEAPIToken={user}!{token_id}={token_secret}'
        }

    def request(self, method, endpoint, **kwargs):
        url = f"{self.base_url}/{endpoint}"
        response = requests.request(
            method, url,
            headers=self.headers,
            verify=False,
            **kwargs
        )
        response.raise_for_status()
        return response.json().get('data')

    def get_nodes(self):
        return self.request('GET', 'nodes')

    def get_vms(self, node):
        return self.request('GET', f'nodes/{node}/qemu')

    def start_vm(self, node, vmid):
        return self.request('POST', f'nodes/{node}/qemu/{vmid}/status/start')

    def stop_vm(self, node, vmid):
        return self.request('POST', f'nodes/{node}/qemu/{vmid}/status/stop')

    def create_vm(self, node, **params):
        return self.request('POST', f'nodes/{node}/qemu', data=params)

    def get_task_status(self, node, upid):
        return self.request('GET', f'nodes/{node}/tasks/{upid}/status')

    def wait_for_task(self, node, upid, timeout=300):
        import time
        start = time.time()
        while time.time() - start < timeout:
            status = self.get_task_status(node, upid)
            if status['status'] == 'stopped':
                return status
            time.sleep(1)
        raise TimeoutError(f"Task {upid} did not complete within {timeout}s")
```

## Advanced Operations

### 1. High Availability (HA)

HA ensures VMs/containers restart on another node if the current node fails.

**HA Resources:**
```bash
# List HA groups
GET /cluster/ha/groups

# Create HA group (nodes that can host the VM)
POST /cluster/ha/groups
Parameters:
- group: Group name (e.g., "production")
- nodes: Node list with priority (e.g., "pve1:1,pve2:2,pve3:1")
- restricted: 1 = only run on these nodes, 0 = can run elsewhere
- nofailback: 1 = stay on current node after recovery

# Delete HA group
DELETE /cluster/ha/groups/{group}
```

**HA Resources (VMs to protect):**
```bash
# List HA managed resources
GET /cluster/ha/resources

# Add VM/container to HA
POST /cluster/ha/resources
Parameters:
- sid: Service ID (e.g., "vm:100" or "ct:200")
- group: HA group name (optional)
- state: "started" or "stopped"
- max_restart: Max restart attempts (default: 1)
- max_relocate: Max relocate attempts (default: 1)

# Remove from HA
DELETE /cluster/ha/resources/{sid}
```

**HA Status and Control:**
```bash
# Get HA status
GET /cluster/ha/status/current

# Request manual migration
POST /cluster/ha/resources/{sid}/migrate
Parameters:
- target: Target node name

# Request relocation (stop on current, start on new)
POST /cluster/ha/resources/{sid}/relocate
Parameters:
- target: Target node name
```

### 2. Live Migration

Migrate running VMs between nodes (requires shared storage or storage migration).

```bash
# Online migrate VM to another node
POST /nodes/{source-node}/qemu/{vmid}/migrate
Parameters:
- target: Target node name (required)
- online: 1 for live migration (VM stays running)
- with-local-disks: 1 to migrate local disks too (requires offline or very recent PVE)

# Check migration status (returns task UPID)
# Poll task status to track progress
```

**Migration Requirements:**
- Shared storage (VM disks accessible from both nodes) OR `with-local-disks=1`
- Same network bridges on both nodes (vmbr0, etc.)
- CPUs compatible (or use `cpu: kvm64` for portability)
- For live migration: Intel-Intel or AMD-AMD (same vendor)

### 3. PCI/GPU Passthrough

Pass physical PCI devices (GPUs, NICs) to VMs.

**Prerequisites:**
- Enable IOMMU in BIOS and kernel
- Load VFIO modules
- Blacklist GPU drivers on host

**Configuration:**
```bash
# Get available PCI devices
GET /nodes/{node}/hardware/pci

# Get PCI device details (includes IOMMU groups)
GET /nodes/{node}/hardware/pci/{pci-id}

# Update VM config with PCI device
POST /nodes/{node}/qemu/{vmid}/config
Parameters:
- hostpci0: PCI device mapping
  Format: "0000:01:00.0" (simple)
  Advanced: "0000:01:00.0,pcie=1,x-vga=1,rombar=0"
  Options:
  - pcie=1: Use PCIe (recommended for modern GPUs)
  - x-vga=1: Primary GPU
  - rombar=0: Hide ROM bar (some GPUs need this)
  - multifunction=1: Pass multifunction device

# Example for NVIDIA GPU
hostpci0: "0000:01:00.0,pcie=1,x-vga=1"

# Example for additional GPU functions (audio)
hostpci1: "0000:01:00.1,pcie=1"

# CPU type must be host for passthrough
cpu: host
```

**USB Passthrough:**
```bash
# List USB devices
GET /nodes/{node}/hardware/usb

# Add to VM config
usb0: "host=1234:5678"  # Vendor:Product ID
usb1: "host=1-2.3"       # Bus-Port path
```

### 4. Backup and Restore

**Backup (vzdump):**
```bash
# Create backup
POST /nodes/{node}/vzdump
Parameters:
- vmid: VM ID(s) to backup (comma-separated or multiple params)
- storage: Target storage ID
- mode: Backup mode
  - snapshot: No downtime, uses QEMU snapshot
  - suspend: Brief pause for consistency
  - stop: Shutdown, backup, start
- compress: Compression level (0-9, default: 1)
- remove: Remove old backups (keep N most recent)
- notes-template: Custom backup notes
- notification-mode: "auto", "always", or "failure"

# Backup all VMs on node
vmid: "all"

# Returns task UPID - poll for completion
```

**Restore:**
```bash
# Restore VM from backup
POST /nodes/{node}/storage/{storage}/content/{backup-file}/restore
Parameters:
- vmid: New VM ID (must be unique)
- force: Overwrite existing VM (dangerous!)
- unique: Generate new MAC addresses (recommended)
- storage: Target storage for disks (can differ from backup)

# Extract config from backup (inspect without restoring)
GET /nodes/{node}/storage/{storage}/content/{backup-file}/config
```

**Backup Management:**
```bash
# List backups in storage
GET /nodes/{node}/storage/{storage}/content?content=backup

# Get backup information
GET /nodes/{node}/storage/{storage}/content/{backup-file}

# Delete backup
DELETE /nodes/{node}/storage/{storage}/content/{backup-file}
```

### 5. Scheduled Jobs

Automate backups and other tasks.

```bash
# List scheduled jobs
GET /cluster/jobs/schedule-analyze

# Create scheduled backup job
POST /cluster/jobs
Parameters:
- id: Job ID (optional, auto-generated)
- type: "vzdump"
- schedule: Cron expression (e.g., "0 2 * * *" for 2 AM daily)
- enabled: 1 or 0
- vmid: VM ID(s) to backup
- storage: Target storage
- mode: "snapshot", "suspend", or "stop"
- compress: 0-9
- remove: Number of backups to keep
- notification-target: Where to send notifications

# Update job
PUT /cluster/jobs/{id}

# Delete job
DELETE /cluster/jobs/{id}

# Get job execution log
GET /cluster/jobs/{id}/log
```

**Cron Expression Examples:**
- `"0 2 * * *"`: Daily at 2 AM
- `"0 0 * * 0"`: Weekly on Sunday midnight
- `"0 */6 * * *"`: Every 6 hours
- `"0 3 1 * *"`: Monthly on 1st at 3 AM

### 6. Permission Management

**Users:**
```bash
# List users
GET /access/users

# Create user
POST /access/users
Parameters:
- userid: User ID (e.g., "john@pam" or "jane@pve")
- password: User password
- groups: Comma-separated group list
- comment: Description
- enable: 1 or 0
- expire: Account expiration (timestamp)

# Update user
PUT /access/users/{userid}

# Delete user
DELETE /access/users/{userid}
```

**Groups:**
```bash
# List groups
GET /access/groups

# Create group
POST /access/groups
Parameters:
- groupid: Group name
- comment: Description

# Delete group
DELETE /access/groups/{groupid}
```

**Roles:**
```bash
# List roles
GET /access/roles

# Create custom role
POST /access/roles
Parameters:
- roleid: Role name
- privileges: Comma-separated list of permissions
  - Common: VM.Allocate, VM.Audit, VM.Config.*, VM.Console, VM.Clone
  - Admin: Administrator (all permissions)
  - NoAccess: No permissions

# Built-in roles: Administrator, PVEAdmin, PVEUser, PVEVMUser, PVETemplateUser, NoAccess
```

**ACLs (Access Control):**
```bash
# List ACLs
GET /access/acl

# Create ACL rule
PUT /access/acl
Parameters:
- path: Resource path (e.g., "/vms/100", "/pool/production", "/")
- roles: Role name(s)
- users: User ID(s) (optional)
- groups: Group name(s) (optional)
- propagate: 1 to apply to children

# Examples:
# Grant user admin on VM 100
path: "/vms/100", users: "john@pam", roles: "Administrator"

# Grant group access to pool
path: "/pool/production", groups: "developers", roles: "PVEVMUser"

# Remove ACL
DELETE /access/acl?path={path}&users={userid}&roles={role}
```

**Pools:**
```bash
# List pools
GET /pools

# Create pool
POST /pools
Parameters:
- poolid: Pool name
- comment: Description

# Update pool (add/remove VMs)
PUT /pools/{poolid}
Parameters:
- vms: "100,101,102" (replaces current list)
- members: Array of objects with type and id

# Delete pool
DELETE /pools/{poolid}
```

### 7. Notifications (Webhooks)

Configure notifications for automation workflows.

```bash
# Get notification endpoints
GET /cluster/notifications/endpoints

# Create webhook endpoint
POST /cluster/notifications/endpoints
Parameters:
- name: Endpoint name
- type: "gotify" or "webhook"
- server: Server URL
- token: Authentication token
- comment: Description
- disable: 0 or 1

# Test notification
POST /cluster/notifications/endpoints/{name}/test

# Get notification targets (who receives what)
GET /cluster/notifications/targets

# Create notification target
POST /cluster/notifications/targets
Parameters:
- name: Target name
- endpoint: Endpoint name
- severity: "info", "notice", "warning", "error", "unknown"

# Get notification matchers (filter rules)
GET /cluster/notifications/matchers
```

**Notification Events:**
- Backup jobs: `vzdump`, `vzrestore`
- Replication: `vzrep`
- HA events: Resource migration, failover
- System: Updates, errors

### 8. Replication

Replicate VM disks to another node for disaster recovery.

```bash
# List replication jobs
GET /nodes/{node}/replication

# Create replication job
POST /nodes/{node}/replication
Parameters:
- id: Job ID (optional)
- target: Target node name
- vmid: VM ID to replicate
- schedule: Cron expression (default: every 15 min)
- rate: Speed limit (MB/s, 0 = unlimited)
- remove_job: Remove job if VM removed

# Get replication status
GET /nodes/{node}/replication/{id}/status

# Run replication now
POST /nodes/{node}/replication/{id}/run_now

# Delete replication job
DELETE /nodes/{node}/replication/{id}
```

### 9. Firewall

Host and VM-level firewall configuration.

**Host Firewall:**
```bash
# Get host firewall rules
GET /nodes/{node}/firewall/rules

# Create host rule
POST /nodes/{node}/firewall/rules
Parameters:
- type: "in" or "out"
- action: "ACCEPT", "DROP", "REJECT"
- macro: Predefined service (e.g., "SSH", "HTTP", "HTTPS")
- source/dest: IP/CIDR
- dport: Destination port(s)
- proto: Protocol (tcp, udp, icmp)
- enable: 0 or 1
- comment: Description

# Get firewall options
GET /nodes/{node}/firewall/options

# Update options
PUT /nodes/{node}/firewall/options
Parameters:
- enable: 0 or 1
- policy_in: Default input policy
- policy_out: Default output policy
```

**VM Firewall:**
```bash
# Get VM firewall rules
GET /nodes/{node}/qemu/{vmid}/firewall/rules

# Create VM rule
POST /nodes/{node}/qemu/{vmid}/firewall/rules
Parameters:
- type, action, macro, source, dest, dport, proto, enable, comment

# Get VM firewall aliases (IP groups)
GET /nodes/{node}/qemu/{vmid}/firewall/aliases

# Create alias
POST /nodes/{node}/qemu/{vmid}/firewall/aliases
Parameters:
- name: Alias name
- cidr: IP/CIDR or IP range

# Get firewall logs
GET /nodes/{node}/qemu/{vmid}/firewall/log
```

**Security Groups:**
```bash
# List security groups
GET /cluster/firewall/groups

# Create security group
POST /cluster/firewall/groups
Parameters:
- group: Group name
- comment: Description

# Add rules to security group
POST /cluster/firewall/groups/{group}
```

### 10. Storage Configuration

**Storage Types:**

| Type | Description | Common Options |
|------|-------------|----------------|
| `dir` | Directory on local filesystem | path, prune-backups |
| `lvm` | LVM logical volumes | vgname, base, safe-remove |
| `lvmthin` | LVM thin provisioning | vgname, thinpool |
| `zfspool` | ZFS pool | pool, sparse, blocksize |
| `nfs` | NFS share | server, export, options |
| `cifs` | SMB/CIFS share | server, share, username, password |
| `iscsi` | iSCSI target | portal, target |
| `cephfs` | CephFS | monhost, fs-name, userid |
| `rbd` | Ceph RBD | monhost, pool, userid |
| `glusterfs` | GlusterFS | server, volume, transport |

**Storage Operations:**
```bash
# List storage
GET /storage

# Get storage configuration
GET /storage/{storage}

# Create storage
POST /storage
Parameters:
- storage: Storage ID
- type: Storage type (dir, lvm, zfspool, nfs, etc.)
- content: Allowed content types (rootdir,images,iso,vztmpl,backup)
- nodes: Nodes that can use this storage
- [type-specific options]

# Update storage
PUT /storage/{storage}

# Delete storage
DELETE /storage/{storage}

# Get storage status
GET /nodes/{node}/storage/{storage}/status
```

### 11. Certificate Management

```bash
# Get current certificate info
GET /nodes/{node}/certificates/acme/certificate

# Get ACME account info
GET /nodes/{node}/certificates/acme/account

# List ACME plugins
GET /cluster/acme/plugins

# Create ACME plugin (for DNS challenge)
POST /cluster/acme/plugins
Parameters:
- id: Plugin name
- type: Plugin type (e.g., "dns_cf" for Cloudflare)
- data: Plugin-specific data (API keys, etc.)

# Order certificate
POST /nodes/{node}/certificates/acme/certificate
Parameters:
- force: 1 to renew even if valid
- restart: 1 to restart proxy after
```

### 12. Node Maintenance

```bash
# Get node subscription info
GET /nodes/{node}/subscription

# Upload subscription key
POST /nodes/{node}/subscription

# Get apt updates available
GET /nodes/{node}/apt/versions

# Run apt update
POST /nodes/{node}/apt/update

# Run apt upgrade
POST /nodes/{node}/apt/upgrade

# Get system log
GET /nodes/{node}/syslog

# Get support dump (diagnostics)
GET /nodes/{node}/report

# Start backup upload to Proxmox (support only)
POST /nodes/{node}/report
```

## References

- [Proxmox VE API Documentation](https://pve.proxmox.com/wiki/Proxmox_VE_API)
- [API Explorer](https://pve.proxmox.com/pve-docs/api-viewer/)
- [pvesh man page](https://pve.proxmox.com/pve-docs/pvesh.1.html)
- [HA Manager](https://pve.proxmox.com/wiki/High_Availability)
- [PCI Passthrough](https://pve.proxmox.com/wiki/PCI_Passthrough)
- [Backup and Restore](https://pve.proxmox.com/wiki/Backup_and_Restore)
- [Firewall](https://pve.proxmox.com/wiki/Firewall)