back
loading skill details...
Visual UI annotation tool for AI agents. Drop the React toolbar into any app — humans click elements and leave feedback, agents receive structured CSS…
agentation — Visual UI Feedback Bridge for AI Agents
The missing link between human eyes and agent code.
Instead of describing "the blue button in the sidebar," you hand the agent .sidebar > button.primary. It can grep for that directly.
When to use this skill
Human needs to point at a UI element and give feedback — without writing selectors
Running iterative UI/UX review cycles between human and coding agent
Building a watch-loop where agent auto-fixes every annotation a human leaves
Capturing CSS selectors, bounding boxes, and React component trees for precise code targeting
Autonomous design critique via agent-browser + self-driving pattern
Integrating visual feedback into agent hooks so annotations auto-appear in agent context
1. Architecture
agentation (monorepo)
├── agentation → npm: agentation (React toolbar component)
│ └── src/index.ts → exports Agentation component + types + utilities
└── agentation-mcp → npm: agentation-mcp (MCP server + CLI)
├── src/cli.ts → agentation-mcp CLI (init, server, doctor)
└── src/server/ → HTTP REST API (port 4747) + SSE events + MCP stdio tools
Two modes of operation:
Mode
How it works
Copy-Paste
Human annotates → clicks Copy → pastes markdown into agent chat
Agent Sync
endpoint prop connects toolbar to MCP server → agent uses agentation_watch_annotations loop
2. Installation
2.1 React Component (toolbar)
npm install agentation -D
# or: pnpm add agentation -D / yarn add agentation -D / bun add agentation -D
Requirements: React 18+, desktop browser, zero runtime deps beyond React (desktop only — no mobile)
🔗 Local-first by design: Annotations are stored locally and auto-sync when connected to the MCP server.
Offline operation — Annotations can be created without a server
Session continuity — Same session persists after page refresh, no duplicates
Agent-first — resolve/dismiss is handled by the agent
2.2 MCP Server — Universal Setup (Recommended)
Fastest method — Auto-detects all installed agents and configures them (Claude Code, Cursor, Codex, Windsurf, and 9+ more agents):
npx add-mcp "npx -y agentation-mcp server"
Or install manually:
npm install agentation-mcp -D
npx agentation-mcp server # HTTP :4747 + MCP stdio
npx agentation-mcp server --port 8080 # custom port
npx agentation-mcp doctor # verify setup
2.3 Claude Code — Official Skill (Minimal Setup)
Recommended for Claude Code users — automatically handles framework detection, package installation, and layout integration:
npx skills add benjitaylor/agentation
# then in Claude Code:
/agentation
3. React Component Setup
Basic (Copy-Paste mode — no server needed)
import { Agentation } from 'agentation';
function App() {
return (
<>
<YourApp />
{process.env.NODE_ENV === 'development' && <Agentation />}
</>
);
}
Next.js App Router
// app/layout.tsx
import { Agentation } from 'agentation';
export default function RootLayout({ children }: { children: React.ReactNode }) {
return (
<html>
<body>
{children}
{process.env.NODE_ENV === 'development' && (
<Agentation endpoint="http://localhost:4747" />
)}
</body>
</html>
);
}
Next.js Pages Router
// pages/_app.tsx
import { Agentation } from 'agentation';
export default function App({ Component, pageProps }) {
return (
<>
<Component {...pageProps} />
{process.env.NODE_ENV === 'development' && (
<Agentation endpoint="http://localhost:4747" />
)}
</>
);
}
Full Props Reference
Prop
Type
Default
Description
endpoint
string
—
MCP server URL for Agent Sync mode
sessionId
string
—
Pre-existing session ID to join
onAnnotationAdd
(a: Annotation) => void
—
Callback when annotation created
onAnnotationDelete
(a: Annotation) => void
—
Callback when annotation deleted
onAnnotationUpdate
(a: Annotation) => void
—
Callback when annotation edited
onAnnotationsClear
(a: Annotation[]) => void
—
Callback when all cleared
onCopy
(markdown: string) => void
—
Callback with markdown on copy
onSubmit
(output: string, annotations: Annotation[]) => void
—
On "Send Annotations" click
copyToClipboard
boolean
true
Set false to suppress clipboard write
onSessionCreated
(sessionId: string) => void
—
Called on new session creation
webhookUrl
string
—
Webhook URL to receive annotation events
4. MCP Server Setup — All Platforms
Fastest method — Universal (auto-detects 9+ agents):
npx add-mcp "npx -y agentation-mcp server"
add-mcp auto-detects Claude Code, Cursor, Codex, Windsurf, and more, writing directly to the correct config.
Start server / verify:
npx agentation-mcp server # HTTP :4747 + MCP stdio
npx agentation-mcp server --port 8080 # custom port
npx agentation-mcp doctor # verify setup
Claude Code (.claude/)
Minimal setup — Official Claude Code Skill (Recommended):
npx skills add benjitaylor/agentation
# In Claude Code:
/agentation
Universal MCP auto-setup (Claude Code + 9+ agents):
npx add-mcp "npx -y agentation-mcp server"
Interactive wizard (Claude Code only):
npx agentation-mcp init
Option A — CLI (recommended):
claude mcp add agentation -- npx -y agentation-mcp server
Option B — config file (~/.claude/claude_desktop_config.json for global, or .claude/mcp.json for project-level):
{
"mcpServers": {
"agentation": {
"command": "npx",
"args": ["-y", "agentation-mcp", "server"]
}
}
}
Interactive wizard (Claude Code only):
npx agentation-mcp init
UserPromptSubmit hook — auto-inject pending annotations on every message.
Add to .claude/settings.json (project) or ~/.claude/settings.json (global):
{
"hooks": {
"UserPromptSubmit": [
{
"type": "command",
"command": "curl -sf --connect-timeout 1 http://localhost:4747/pending 2>/dev/null | python3 -c \"import sys,json;d=json.load(sys.stdin);c=d['count'];exit(0)if c==0 else[print(f'\\n=== AGENTATION: {c} UI annotations ===\\n'),*[print(f\\\"[{i+1}] {a['element']} ({a['elementPath']})\\n {a['comment']}\\n\\\")for i,a in enumerate(d['annotations'])],print('=== END ===\\n')]\" 2>/dev/null;exit 0"
}
]
}
}
Codex CLI (~/.codex/)
Add to ~/.codex/config.toml:
# Agentation MCP Server
[[mcp_servers]]
name = "agentation"
command = "npx"
args = ["-y", "agentation-mcp", "server"]
# Optional: teach Codex about watch-loop
developer_instructions = """
When user says "watch mode" or "agentation watch", call agentation_watch_annotations in a loop.
For each annotation: acknowledge it, fix the code using the elementPath CSS selector, resolve with summary.
"""
Restart Codex CLI after editing config.toml.
Gemini CLI (~/.gemini/)
Option A — CLI:
gemini mcp add agentation npx -y agentation-mcp server
# or with explicit scope
gemini mcp add -s user agentation npx -y agentation-mcp server
Option B — config file (~/.gemini/settings.json for global, .gemini/settings.json for project):
{
"mcpServers": {
"agentation": {
"command": "npx",
"args": ["-y", "agentation-mcp", "server"]
}
}
}
AfterAgent hook — trigger annotation check after each agent turn:
{
"mcpServers": {
"agentation": {
"command": "npx",
"args": ["-y", "agentation-mcp", "server"]
}
},
"hooks": {
"AfterAgent": [
{
"matcher": "",
"hooks": [
{
"type": "command",
"command": "curl -sf --connect-timeout 1 http://localhost:4747/pending 2>/dev/null | python3 -c \"import sys,json;d=json.load(sys.stdin);c=d.get('count',0);[print(f'[agentation] {c} pending annotations'),exit(1)]if c>0 else exit(0)\" 2>/dev/null;exit 0",
"description": "Check for pending agentation annotations"
}
]
}
]
}
}
OpenCode (~/.config/opencode/)
Add to ~/.config/opencode/opencode.json:
{
"mcp": {
"agentation": {
"type": "local",
"command": ["npx", "-y", "agentation-mcp", "server"]
}
}
}
With environment variables:
{
"mcp": {
"agentation": {
"type": "local",
"command": ["npx", "-y", "agentation-mcp", "server"],
"environment": {
"AGENTATION_STORE": "sqlite",
"AGENTATION_EVENT_RETENTION_DAYS": "7"
}
}
}
}
Restart OpenCode after editing. MCP tools (agentation_*) will be available immediately.
Universal (npx add-mcp)
Works for any MCP-compatible agent:
npx add-mcp "npx -y agentation-mcp server"
Quick-Setup Script
Save and run bash setup-agentation-mcp.sh [--all | --claude | --codex | --gemini | --opencode]:
#!/usr/bin/env bash
# setup-agentation-mcp.sh — Register agentation MCP for all agent platforms
set -euo pipefail
SETUP_CLAUDE=false; SETUP_CODEX=false; SETUP_GEMINI=false; SETUP_OPENCODE=false
while [[ $# -gt 0 ]]; do
case "$1" in
--claude) SETUP_CLAUDE=true ;;
--codex) SETUP_CODEX=true ;;
--gemini) SETUP_GEMINI=true ;;
--opencode) SETUP_OPENCODE=true ;;
--all) SETUP_CLAUDE=true; SETUP_CODEX=true; SETUP_GEMINI=true; SETUP_OPENCODE=true ;;
esac
shift
done
[[ "$SETUP_CLAUDE$SETUP_CODEX$SETUP_GEMINI$SETUP_OPENCODE" == "falsefalsefalsefalse" ]] && \
SETUP_CLAUDE=true && SETUP_CODEX=true && SETUP_GEMINI=true && SETUP_OPENCODE=true
MCP_JSON='"agentation": {"command": "npx", "args": ["-y", "agentation-mcp", "server"]}'
# Claude Code
if [[ "$SETUP_CLAUDE" == "true" ]]; then
mkdir -p ~/.claude
CFG=~/.claude/claude_desktop_config.json
if [[ -f "$CFG" ]] && command -v jq &>/dev/null; then
jq ".mcpServers += {$MCP_JSON}" "$CFG" > "$CFG.tmp" && mv "$CFG.tmp" "$CFG"
else
echo "{\"mcpServers\": {$MCP_JSON}}" > "$CFG"
fi
echo "✅ Claude Code: $CFG"
fi
# Codex CLI
if [[ "$SETUP_CODEX" == "true" ]]; then
mkdir -p ~/.codex
CFG=~/.codex/config.toml
if ! grep -q "agentation" "$CFG" 2>/dev/null; then
printf '\n[[mcp_servers]]\nname = "agentation"\ncommand = "npx"\nargs = ["-y", "agentation-mcp", "server"]\n' >> "$CFG"
fi
echo "✅ Codex CLI: $CFG"
fi
# Gemini CLI
if [[ "$SETUP_GEMINI" == "true" ]]; then
mkdir -p ~/.gemini
CFG=~/.gemini/settings.json
if [[ -f "$CFG" ]] && command -v jq &>/dev/null; then
jq ".mcpServers += {$MCP_JSON}" "$CFG" > "$CFG.tmp" && mv "$CFG.tmp" "$CFG"
else
echo "{\"mcpServers\": {$MCP_JSON}}" > "$CFG"
fi
echo "✅ Gemini CLI: $CFG"
fi
# OpenCode
if [[ "$SETUP_OPENCODE" == "true" ]]; then
mkdir -p ~/.config/opencode
CFG=~/.config/opencode/opencode.json
ENTRY='"agentation": {"type": "local", "command": ["npx", "-y", "agentation-mcp", "server"]}'
if [[ -f "$CFG" ]] && command -v jq &>/dev/null; then
jq ".mcp += {$ENTRY}" "$CFG" > "$CFG.tmp" && mv "$CFG.tmp" "$CFG"
else
echo "{\"mcp\": {$ENTRY}}" > "$CFG"
fi
echo "✅ OpenCode: $CFG"
fi
echo ""
echo "Done. Restart your agent(s) and run: npx agentation-mcp server"
5. MCP Tools (Agent API)
Tool
Parameters
Description
agentation_list_sessions
none
List all active annotation sessions
agentation_get_session
sessionId: string
Get session with all annotations
agentation_get_pending
sessionId: string
Get pending annotations for a session
agentation_get_all_pending
none
Get pending annotations across ALL sessions
agentation_acknowledge
annotationId: string
Mark annotation as acknowledged (agent is working on it)
agentation_resolve
annotationId: string, summary?: string
Mark as resolved with optional summary
agentation_dismiss
annotationId: string, reason: string
Dismiss with required reason
agentation_reply
annotationId: string, message: string
Add reply to annotation thread
agentation_watch_annotations
sessionId?: string, batchWindowSeconds?: number (default 10, max 60), timeoutSeconds?: number (default 120, max 300)
Block until new annotations arrive — core watch-loop tool
6. Workflow Patterns
Pattern 1: Copy-Paste (Simplest, No Server)
1. Human opens app in browser
2. Clicks agentation toolbar → activates
3. Clicks element → adds comment → clicks Copy
4. Pastes markdown output into agent chat
5. Agent receives CSS selectors, elementPath, boundingBox
6. Agent greps/edits code using selector
Pattern 2: MCP Watch Loop (Recommended for iterative review)
Agent: agentation_watch_annotations (blocks up to 120s)
→ Human adds annotation in browser
→ Agent receives batch immediately
→ Agent: agentation_acknowledge(annotationId)
→ Agent makes code changes using elementPath as grep target
→ Agent: agentation_resolve(annotationId, "Changed button color to #3b82f6")
→ Agent: agentation_watch_annotations (loops again)
CLAUDE.md / GEMINI.md / Codex developer_instructions — add for automated watch:
When I say "watch mode" or "agentation watch", call agentation_watch_annotations in a loop.
For each annotation received:
1. Call agentation_acknowledge(annotationId)
2. Use elementPath to locate the code: Grep(elementPath) or search codebase for CSS class
3. Make the minimal change described in the comment
4. Call agentation_resolve(annotationId, "<brief summary of what was changed>")
Continue watching until I say stop, or until timeout.
Pattern 3: Platform-Specific Hook (Passive Injection)
The hook from Section 4 auto-appends pending annotations to every agent message — no "watch mode" needed. Works across all platforms.
Pattern 4: Autonomous Self-Driving Critique
Two-agent setup for fully autonomous UI review cycles:
Session 1 (Critic — uses agent-browser):
# Start headed browser pointing at your dev server
agent-browser open http://localhost:3000
agent-browser snapshot -i
# Agent navigates, clicks elements via agentation toolbar, adds critique
# Annotations flow to agentation MCP server automatically
Session 2 (Fixer — watches MCP):
agentation_watch_annotations → receives critique → acknowledge → edit → resolve → loop
Pattern 5: Webhook Integration
<Agentation webhookUrl="https://your-server.com/webhook" />
# or env var:
# AGENTATION_WEBHOOK_URL=https://your-server.com/webhook
7. Annotation Type (Full Schema)
type Annotation = {
// Core
id: string;
x: number; // % of viewport width (0-100)
y: number; // px from document top
comment: string; // User's feedback text
element: string; // Tag name: "button", "div", etc.
elementPath: string; // CSS selector: "body > main > button.cta" ← grep target
timestamp: number;
// Context
selectedText?: string;
boundingBox?: { x: number; y: number; width: number; height: number };
nearbyText?: string;
cssClasses?: string;
nearbyElements?: string;
computedStyles?: string;
fullPath?: string;
accessibility?: string;
reactComponents?: string; // "App > Dashboard > Button" ← component grep target
isMultiSelect?: boolean;
isFixed?: boolean;
// Lifecycle (server-synced)
sessionId?: string;
url?: string;
intent?: "fix" | "change" | "question" | "approve";
severity?: "blocking" | "important" | "suggestion";
status?: "pending" | "acknowledged" | "resolved" | "dismissed";
thread?: ThreadMessage[];
createdAt?: string;
updatedAt?: string;
resolvedAt?: string;
resolvedBy?: "human" | "agent";
};
Annotation lifecycle:
pending → acknowledged → resolved
↘ dismissed (requires reason)
8. HTTP REST API (port 4747)
# Sessions
POST /sessions # Create session
GET /sessions # List all sessions
GET /sessions/:id # Get session + annotations
# Annotations
POST /sessions/:id/annotations # Add annotation
GET /annotations/:id # Get annotation
PATCH /annotations/:id # Update annotation
DELETE /annotations/:id # Delete annotation
GET /sessions/:id/pending # Pending for session
GET /pending # ALL pending across sessions
# Events (SSE streaming)
GET /sessions/:id/events # Session stream
GET /events # Global stream (?domain=filter)
# Health
GET /health
GET /status
9. Environment Variables
Variable
Description
Default
AGENTATION_STORE
memory or sqlite
sqlite
AGENTATION_WEBHOOK_URL
Single webhook URL
—
AGENTATION_WEBHOOKS
Comma-separated webhook URLs
—
AGENTATION_EVENT_RETENTION_DAYS
Days to keep events
7
SQLite storage: ~/.agentation/store.db
10. Programmatic Utilities
import {
identifyElement, identifyAnimationElement,
getElementPath, getNearbyText, getElementClasses,
isInShadowDOM, getShadowHost, closestCrossingShadow,
loadAnnotations, saveAnnotations, getStorageKey,
type Annotation, type Session, type ThreadMessage,
} from 'agentation';
11. Platform Support Matrix
Platform
Config File
MCP Key
Hook
Claude Code
~/.claude/claude_desktop_config.json
mcpServers
hooks.UserPromptSubmit in settings.json
Codex CLI
~/.codex/config.toml
[[mcp_servers]] (TOML)
developer_instructions + notify
Gemini CLI
~/.gemini/settings.json
mcpServers
hooks.AfterAgent in settings.json
OpenCode
~/.config/opencode/opencode.json
mcp (type: "local")
Skills system (no hook needed)
Cursor / Windsurf
.cursor/mcp.json / .windsurf/mcp.json
mcpServers
—
Best practices
Always gate <Agentation> with NODE_ENV === 'development' — never ship to production
Use MCP watch-loop over copy-paste for iterative cycles — eliminates context switching
Call agentation_acknowledge immediately when starting a fix — signals human
Include a summary in agentation_resolve — gives human traceability
Process severity: "blocking" annotations first in the watch loop
Use elementPath as the primary grep/search target in code — it's a valid CSS selector
Use reactComponents field when the codebase is React — matches component names directly
Add the appropriate hook for your platform (Section 4) for zero-friction passive injection
For autonomous self-driving, use agent-browser in headed mode with agentation mounted
12. jeo Integration (annotate keyword)
agentation integrates as the VERIFY_UI phase of the jeo skill.
This follows the same pattern as plannotator operating in planui / ExitPlanMode.
annotate is the primary keyword. agentui is kept as a backward-compatible alias.
How it works
plannotator (planui): agentation (annotate):
Write plan.md Mount <Agentation> in app UI
↓ blocking ↓ blocking
Run plannotator agentation_watch_annotations
↓ ↓
Approve/Feedback in UI Create annotation in UI
↓ ↓
Confirm approved:true annotation ack→fix→resolve
↓ ↓
Enter EXECUTE Next step or loop
Trigger Keywords
Keyword
Platform
Action
annotate
Claude Code
agentation_watch_annotations MCP blocking call
annotate
Codex
ANNOTATE_READY signal → jeo-notify.py HTTP polling
annotate
Gemini
GEMINI.md instruction: HTTP REST polling pattern
/jeo-annotate
OpenCode
opencode.json mcp.agentation + instructions
agentui (deprecated)
All platforms
Same behavior as above — backward-compatible alias
UI review
All platforms
Same as annotate
Using with jeo
# 1. agentation auto-registered when installing jeo
bash .agent-skills/jeo/scripts/install.sh --with-agentation
# Or full install:
bash .agent-skills/jeo/scripts/install.sh --all
# 2. Mount agentation component in app
# app/layout.tsx or pages/_app.tsx:
# <Agentation endpoint="http://localhost:4747" />
# 3. Start MCP server
npx agentation-mcp server
# 4. Enter annotate keyword in agent → watch loop starts (agentui also works as backward-compatible alias)
# Claude Code: direct MCP tool call
# Codex: output ANNOTATE_READY (or AGENTUI_READY) → notify hook auto-polls
# Gemini: GEMINI.md HTTP polling pattern
# OpenCode: /jeo-annotate slash command (or /jeo-agentui — deprecated)
Separation from plannotator (Phase Guard)
plannotator and agentation use the same blocking loop pattern but only operate in different phases:
Tool
Allowed phase
Hook Guard
plannotator
plan only
jeo-state.json → phase === "plan"
agentation
verify / verify_ui only
jeo-state.json → phase === "verify_ui"
Each platform's hook script checks the phase field in jeo-state.json to prevent execution in the wrong phase.
Without this guard, both tools could run simultaneously in Gemini's AfterAgent hook.
Pre-flight Check
3-step check before entering VERIFY_UI:
Server status: GET /health — whether agentation-mcp server is running
Session exists: GET /sessions — whether <Agentation> component is mounted
Pending annotations: GET /pending — number of annotations to process
After passing, set phase in jeo-state.json to "verify_ui" and agentation.active to true.
Loop Verification Test
# Run agentation watch loop integration test
bash .agent-skills/agentation/scripts/verify-loop.sh
# Quick test (skip error cases)
bash .agent-skills/agentation/scripts/verify-loop.sh --quick
4-step verification: Server Health → Annotation CRUD → ACK-RESOLVE Cycle → Error Cases
Evaluation Flow (jeo VERIFY_UI phase)
jeo "<task>"
│
[1] PLAN (plannotator loop) ← approve plan.md
[2] EXECUTE (team/bmad)
[3] VERIFY
├─ agent-browser snapshot
├─ Pre-flight check (server + session + pending)
└─ annotate → VERIFY_UI (agentation loop) ← this phase (agentui also backward-compatible)
├─ ACK → FIND → FIX → RESOLVE
├─ RE-SNAPSHOT (agent-browser) ← re-check after fix
└─ update agentation fields in jeo-state.json
[4] CLEANUP
For detailed jeo integration: see jeo SKILL.md Section 3.3.1 detailed workflow
References
agentation repo
agentation npm
agentation-mcp npm
Gemini CLI MCP docs
agent-browser skill
Metadata
Version: 1.1.0
Source: benjitaylor/agentation (PolyForm Shield 1.0.0)
Packages: agentation@2.2.1, agentation-mcp@1.2.0
Last updated: 2026-03-05
Scope: UI annotation bridge for human-agent feedback loops — Claude Code, Codex, Gemini CLI, OpenCodedon't have the plugin yet? install it then click "run inline in claude" again.