KylinOS Computer Use

SKILL.md

---
name: kylinos-computer-use
description: Use wlcctrl on KylinOS V11 Desktop / UKUI / wlcom Wayland desktops for fast GUI automation tests: find windows/outputs, launch or focus apps, screenshot, click/type/drag, handle scale-aware coordinates, and verify GUI results.
---

# KylinOS Computer Use

Use for GUI automation on KylinOS V11 Desktop with wlcom `wlcctrl`.

## Fast path

Resolve script paths relative to this skill directory.

```bash
# Inspect once per task
scripts/wlcctrl-info.sh

# Launch/focus app, then find target window
# For Kaiming apps, prefer: kaiming list | grep -i '<app>'; kaiming run <app-id>
wlcctrl --search '<app_id_or_title_regex>'
wlcctrl --getwindowgeometry <w_uuid>

# Capture window for coordinate picking; keep the user's window position by default
wlcctrl --windowcapture <w_uuid> --path /tmp/window.png

# Keyboard is fastest when available
wlcctrl --windowactivate <w_uuid>
wlcctrl --keystring 'text'
wlcctrl --key ctrl+s

# Click a point from window-screenshot coordinates when keyboard is insufficient
scripts/wlcctrl-window-click.sh <w_uuid> <relative_x> <relative_y> [button]
```

Do **not** move a window to `(0,0)` as a default setup step. Use `scripts/wlcctrl-move-window.sh` only when the test specifically requires a known position, a window is partly off-screen, or the user asks for it.

## Rules to remember

- `--mousemove` is absolute, not relative.
- On tested V11 machines, `--mousemove` input = logical coordinate × output scale; `--getmouselocation` returns logical coordinates.
- `--windowmove -x/-y` uses logical coordinates directly, but avoid moving windows unless needed.
- Use `wlcctrl --outputs` + `--getdisplaygeometry <o_uuid>` to check output UUID, position, mode, and scale.
- For wheel/spinner controls, prefer mouse drag with `--mousepress`/`--mousemove`/`--mouserelease`; `--scroll` may jump multiple values.
- Current `wlcctrl v1.0.0` help/man shows screenshots but no recording subcommand.
- Avoid `--windowclose` / `--windowkill` unless explicitly requested.

## Core commands

```bash
wlcctrl --outputs
wlcctrl --getdisplaygeometry <o_uuid>
wlcctrl --list
wlcctrl --search '<regex>'
wlcctrl --getactivewindow
wlcctrl --getwindowname <w_uuid>
wlcctrl --getwindowgeometry <w_uuid>
wlcctrl --windowactivate <w_uuid>
wlcctrl --windowmove <w_uuid> -x <x> -y <y>
wlcctrl --windowcapture <w_uuid> --path /tmp/window.png
wlcctrl --fullscreencapture --path /tmp/full.png
wlcctrl --setarea <x,y,width,height> --path /tmp/area.png
wlcctrl --mousemove <x>,<y>
wlcctrl --getmouselocation
wlcctrl --mousebutton 1
wlcctrl --mousepress 1 --mouserelease 1
wlcctrl --scroll -y <delta>
wlcctrl --keystring 'text'
wlcctrl --key ctrl+s
```

## Coordinate formula

Given window geometry `(WX,WY)`, output scale `S`, and target `(RX,RY)` inside a window screenshot:

```text
logical_x = WX + RX
logical_y = WY + RY
mousemove_x = round(logical_x * S)
mousemove_y = round(logical_y * S)
```

Then verify with `wlcctrl --getmouselocation` before clicking.

## Speed tips

- Cache output scale and window geometry for a task; repeated `wlcctrl` calls are relatively expensive.
- Batch simple command sequences in one shell script, but keep screenshot/vision checkpoints only where they change the next decision.
- For Kaiming-packaged apps, launch by app ID with `kaiming run <app-id>` instead of executing wrapper symlinks directly.

## When more detail is needed

Read `references/details.md` for examples, multi-step test pattern, spinner controls, screenshots, and caveats.