Desktop Automation Pro

SKILL.md

---
name: desktop-automation-pro
description: |
  Desktop GUI automation toolkit for browser, mobile devices, and native applications.
  桌面 GUI 自动化工具包，支持浏览器、移动设备和原生应用。
  
  Use this skill when:
  使用此技能的场景：
  - Automating web browsers (Chromium-based)
  - 控制 Chromium 浏览器
  - Controlling paired mobile devices (Android/iOS/macOS)
  - 控制配对的移动设备
  - Taking screenshots of desktop/windows/regions
  - 桌面/窗口/区域截图
  - Automating Windows native applications via Python
  - 通过 Python 自动化 Windows 原生应用
  - Mouse/keyboard simulation
  - 鼠标键盘模拟
  
  Triggers: 桌面自动化, GUI自动化, 浏览器自动化, 手机控制, 截图, desktop automation, GUI automation, browser automation, mobile control, screenshot, pyautogui, pywinauto
---

# Desktop GUI Automation / 桌面 GUI 自动化

Cross-platform GUI automation toolkit with browser control, mobile device management, and native app automation capabilities.

跨平台 GUI 自动化工具包，支持浏览器控制、移动设备管理和原生应用自动化。

## Capabilities Overview / 能力概览

| Platform 平台 | Tool 工具 | Use Case 使用场景 |
|---------------|-----------|-------------------|
| Browser 浏览器 | `browser` | Web automation, screenshots, form filling |
| Mobile 移动设备 | `nodes` | Device control, screen recording, camera |
| Desktop 桌面 | `screenshot` skill | Window/region capture |
| Native 原生应用 | Python scripts | Windows app automation, mouse/keyboard simulation |

---

## 1. Browser Automation / 浏览器自动化

Use the `browser` tool for Chromium-based browser control.

使用 `browser` 工具控制 Chromium 浏览器。

### Common Actions / 常用操作

```markdown
# Open URL / 打开网址
browser action=open url="https://example.com"

# Take screenshot / 截图
browser action=screenshot

# Click element / 点击元素
browser action=act kind=click ref="e12"

# Type text / 输入文本
browser action=act kind=type text="Hello" ref="e15"

# Get page snapshot / 获取页面快照
browser action=snapshot
```

### Workflow Example / 工作流示例

1. Start browser: `browser action=start`
2. Navigate: `browser action=open url="..."`
3. Snapshot to find elements: `browser action=snapshot`
4. Interact: `browser action=act kind=click ref="..."`
5. Capture result: `browser action=screenshot`

---

## 2. Mobile Device Control / 移动设备控制

Use the `nodes` tool for paired devices (Android/iOS/macOS).

使用 `nodes` 工具控制配对设备。

### Available Actions / 可用操作

| Action 操作 | Description 描述 |
|-------------|------------------|
| `camera_snap` | Take photo / 拍照 |
| `screen_record` | Record screen / 录屏 |
| `location_get` | Get GPS location / 获取位置 |
| `device_info` | Device status / 设备状态 |
| `run` | Execute command (iOS/macOS) / 执行命令 |

### Example Usage / 使用示例

```markdown
# List paired devices / 列出配对设备
nodes action=status

# Take screenshot / 截图
nodes action=camera_snap node="my-iphone" facing="back"

# Record screen / 录屏
nodes action=screen_record node="my-android" durationMs=10000
```

---

## 3. Native App Automation / 原生应用自动化

For Windows native applications, use Python scripts via `exec`.

Windows 原生应用通过 Python 脚本执行。

### Option A: Mouse/Keyboard Simulation / 鼠标键盘模拟

Use `pyautogui` for global input simulation.

使用 `pyautogui` 进行全局输入模拟。

```python
# scripts/pyautogui_demo.py
import pyautogui
import time

# Safety: move mouse to corner to abort / 安全：移到角落中止
pyautogui.FAILSAFE = True

# Get screen size / 获取屏幕尺寸
width, height = pyautogui.size()

# Move mouse / 移动鼠标
pyautogui.moveTo(100, 200, duration=0.5)

# Click / 点击
pyautogui.click(x=100, y=200)

# Type text / 输入文本
pyautogui.write('Hello World', interval=0.1)

# Hotkey / 快捷键
pyautogui.hotkey('ctrl', 'c')

# Screenshot / 截图
screenshot = pyautogui.screenshot()
screenshot.save('screenshot.png')

# Locate image on screen / 在屏幕上定位图片
position = pyautogui.locateOnScreen('button.png')
if position:
    pyautogui.click(position)
```

### Option B: Windows App Control / Windows 应用控制

Use `pywinauto` for native Windows application control.

使用 `pywinauto` 控制 Windows 原生应用。

```python
# scripts/pywinauto_demo.py
from pywinauto import Application
import time

# Method 1: Start new app / 方式1：启动新应用
app = Application().start('notepad.exe')

# Method 2: Connect to existing app / 方式2：连接已有应用
# app = Application().connect(title='Untitled - Notepad')

# Get main window / 获取主窗口
win = app.window(title='Untitled - Notepad')

# Type text / 输入文本
win.Edit.type_keys('Hello World{ENTER}', with_spaces=True)

# Menu operations / 菜单操作
win.menu_select('File->Save')

# Click button / 点击按钮
# win.Button1.click()

# Close window / 关闭窗口
win.close()
```

### Installation / 安装依赖

```bash
pip install pyautogui pywinauto pillow
```

---

## 4. Screenshot Capture / 截图捕获

Use the `screenshot` skill for desktop capture.

使用 `screenshot` 技能进行桌面截图。

```markdown
# Full screen / 全屏
Use screenshot skill

# Specific window / 特定窗口
Use browser or nodes tool

# Region / 区域
Use pyautogui.screenshot(region=(x, y, w, h))
```

---

## Quick Reference / 快速参考

| Task 任务 | Recommended Tool 推荐工具 |
|-----------|---------------------------|
| Web form filling 网页表单填充 | `browser` |
| Web scraping 网页抓取 | `browser` + snapshot |
| Mobile screen record 手机录屏 | `nodes` |
| Windows app control Windows应用控制 | `pywinauto` script |
| Global mouse/keyboard 全局鼠标键盘 | `pyautogui` script |
| Desktop screenshot 桌面截图 | `screenshot` skill |

---

## Best Practices / 最佳实践

1. **Browser**: Always use `snapshot` before interacting to get fresh element refs.
   浏览器：交互前先 `snapshot` 获取最新元素引用。

2. **Mobile**: Check device status with `nodes action=status` first.
   移动设备：先检查设备状态。

3. **Native Apps**: Enable FAILSAFE for pyautogui; add delays between actions.
   原生应用：启用 FAILSAFE 安全机制；操作间添加延迟。

4. **Error Handling**: Use try-except and validate element presence before clicking.
   错误处理：使用 try-except，点击前验证元素存在。

---

## Troubleshooting / 故障排查

| Issue 问题 | Solution 解决方案 |
|------------|-------------------|
| Browser won't start 浏览器无法启动 | Check if Chrome/Edge is installed |
| Device not found 设备未找到 | Run `nodes action=status` to check pairing |
| pyautogui click misses 点击偏移 | Check DPI scaling; use `pyautogui.position()` to verify |
| pywinauto connection failed 连接失败 | Run as admin; check app window title |