long-run-harness

SKILL.md

---
name: long-run-harness
description: >
  Use when building a Planner→Generator→Evaluator multi-agent harness or long-running
  orchestrator. Triggers: "build a harness", "multi-agent pipeline", "agent loop",
  "automate app building with agents", "GAN-style agent system", "sprint-based agent
  workflow", "I want agents to plan, build, and evaluate automatically".
  NOT for: asking Codex to build an app directly, single-file edits, pure API usage questions.
---

# Long-Running App Harness — SDK Implementation

Produces a runnable harness that orchestrates Claude SDK agents, with optional Codex CLI
or DeepSeek-backed `deepcode` SDK backends for selected roles.
**You are writing the harness, not running inside it.**

Use `query()` + `ClaudeAgentOptions` for agentic loops; `tool()` + `create_sdk_mcp_server()`
for structured output.

Default to SDK-only provider access. Direct provider clients are allowed only behind a small
adapter when the SDK does not expose a needed capability (for example, image/vision scoring);
put the adapter behind config and keep the rest of the harness SDK-based.

```
uv venv
uv pip install claude-agent-sdk pyyaml
```

Harness template structure:
```
harness/
  harness.py; config.yaml; config.py; log.py
  agents/ planner.py; generator.py; evaluator.py
  models/ state.py
  prompts/ planner.md; generator.md; evaluator.md
```

Run output structure:
```
project_dir/
  src/                         # ALL Generator-created app code lives here
  harness-state/
    spec.md                    # Planner output
    sprints.md                 # Human-readable sprint plan / scope
    contracts/ contract-sprint-N.json
    handoffs/ handoff-sprint-N.json
    evals/ eval-sprint-N-iter-M.json
    evidence/
      sprint-N/
        screenshots/
        axe/
        lighthouse/
        browser/
        artifacts/
    tmp/                       # disposable run files; cleaned by retention policy
  harness-logs/ run-YYYYMMDD-HHMMSS.log
```

**Clean workspace boundary:** Harness state, logs, screenshots, test output, browser traces,
Lighthouse reports, axe reports, temporary manifests, and evaluation artifacts stay under
`harness-state/` or `harness-logs/`. They do not go in `src/`, `public/`, root docs, or app
routes unless the sprint contract explicitly declares a public evidence surface and a cleanup
plan.

**Default hard boundary:** For greenfield app generation, Generator file tools and Codex cwd
point at `project_dir/src`, not `project_dir`.

**Existing-codebase exception:** For maintenance or production-hardening harnesses over an
existing repo, Generator may need repo-root cwd. In that case you MUST define write allowlists,
protected paths, artifact directories, and git safety rules before generation starts.
**Load:** `$SKILL_DIR/instructions/mode-selection.md`
**Load:** `$SKILL_DIR/instructions/workspace-hygiene.md`

---

## Routing

| User Signal | Route |
|---|---|
| "build a harness / pipeline" | Start at Phase 1 |
| "add an evaluator" | Jump to Phase 4 |
| "add state / handoff" | Jump to Phase 5 |
| "looping forever / broken" | Check feedback loop termination in Phase 5 |
| "just explain what a harness does" | Explain concept, don't write code |

---

## Phase 1: Design the Harness

**Load:** `$SKILL_DIR/instructions/planner-questions.md`

**⚠️ HARD GATE:** Ask the design questions. Get answers to 1–3 before writing any code:
1. What does the harness build? (sets Generator tools + Evaluator rubric)
2. Python or TypeScript? (default: Python)
3. Backend + model per agent? (default: all `claude`; all choices → `config.yaml`)
   Ask: planner/generator/evaluator — `claude`, `codex`, or `deepcode`?
   - If `claude`: which model? Thinking enabled?
   - If `codex`: which model + `reasoning_effort`?
   - If `deepcode`: which model? Confirm `/Users/xzhao/.local/bin/deepcode` and `DEEPSEEK_API_KEY`.
4. Harness mode:
   - `greenfield`: create a new app under `project_dir/src`
   - `existing-codebase`: modify an existing repo
   - `production-qa`: mostly evaluate/build/test and generate targeted fix sprints
5. Artifact policy: where should screenshots, logs, eval JSON, build output, browser traces,
   temp files, and evidence manifests go? Default: `harness-state/evidence/` and
   `harness-logs/`; never scatter them through the repo.

Create skeleton:
```bash
mkdir -p harness/agents harness/models harness/prompts
touch harness/harness.py harness/log.py harness/agents/__init__.py harness/models/__init__.py
```

At runtime, create the project folders before agents run:
```python
APP_DIR = PROJECT_DIR / "src"
STATE_DIR = PROJECT_DIR / "harness-state"
LOG_DIR = PROJECT_DIR / "harness-logs"
for path in [
    APP_DIR,
    STATE_DIR / "contracts",
    STATE_DIR / "handoffs",
    STATE_DIR / "evals",
    STATE_DIR / "evidence",
    STATE_DIR / "tmp",
    LOG_DIR,
]:
    path.mkdir(parents=True, exist_ok=True)
```

For existing-codebase mode, `APP_DIR` is usually `PROJECT_DIR`, but artifact paths stay the
same and write guards become mandatory.

**`config.yaml` + `config.py`** — all tunable parameters here; never hardcode in agent files.
**Load:** `$SKILL_DIR/instructions/config.md` for the full `HarnessConfig` dataclass.
```python
cfg = HarnessConfig.load(Path(__file__).parent / "config.yaml")
# Always: cfg.agents.generator.model  — never: "claude-opus-4-7"
# Backend:  cfg.agents.generator.backend   ("claude" | "codex" | "deepcode")
# Thinking: cfg.agents.evaluator.thinking.enabled / .budget_tokens
# Codex:    cfg.agents.generator.codex.reasoning_effort
# Deepcode: cfg.agents.generator.deepcode.cli_path / .env
# Workspace: cfg.workspace.mode / cfg.workspace.artifact_root / cfg.workspace.write_allowlist
```

**`models/state.py`** — write first; all other files import from it.
**Load:** `$SKILL_DIR/instructions/context-handoff.md` (`HandoffState`, `EvalResult`, `format_handoff_for_prompt`).
**Load:** `$SKILL_DIR/instructions/sprint-contracts.md` (`SprintContract` + negotiation protocol).

**`log.py`** — dual stdout + timestamped file under `harness-logs/`.
**Load:** `$SKILL_DIR/instructions/logging.md` for full implementation.
```python
log.setup(PROJECT_DIR, label="run")  # once in main()
logger = log.get()                   # in every agent
```

---

## Phase 2: Planner Agent

**Load:** `$SKILL_DIR/instructions/planner-questions.md` for system prompt template.
**Load:** `$SKILL_DIR/instructions/agent-patterns.md` for full `run_planner` implementation.

`run_planner(brief, session_id, cfg)` → `(reply, new_session_id)`.
`ClaudeAgentOptions(resume=session_id)` continues session without resending history.

```python
def extract_sprint_plan(spec: str) -> str:
    marker = "## Sprint Definitions"
    return spec[spec.find(marker):].strip() if marker in spec else spec.strip()

spec, session_id = "", None
while "SPEC_COMPLETE" not in spec:
    user_input = input("[Planner asks]: ").strip() if session_id else initial_brief
    spec, session_id = run_planner(user_input, session_id, cfg)
STATE_DIR.joinpath("spec.md").write_text(spec.replace("SPEC_COMPLETE", "").strip())
STATE_DIR.joinpath("sprints.md").write_text(extract_sprint_plan(spec))
```

`sprints.md` is the human-readable sprint plan copied or derived from the SPEC's sprint
definitions. Confirmed executable contracts are stored separately as JSON under
`harness-state/contracts/`.

---

## Phase 3: Generator Agent

**Load:** `$SKILL_DIR/instructions/agent-patterns.md` for `run_generator` + `self_assess` implementations.
**If mode is not greenfield:** load `$SKILL_DIR/instructions/git-safety.md`.

```python
def run_generator(
    spec, contract, project_dir,
    handoff=None, strategic_framing=None, cfg=None,
) -> str: ...

ClaudeAgentOptions(
    model=cfg.agents.generator.model,
    allowed_tools=["Write", "Read", "Edit", "Bash", "Glob"],
    cwd=str(project_dir / "src"), permission_mode="bypassPermissions",
)
```

The Generator must treat `project_dir/src` as the app root. It may read harness state
through the prompt, but it must not create code, package files, or tests outside `src`.

In existing-codebase mode, replace that rule with:
- cwd may be repo root.
- Generator may edit only paths matching `cfg.workspace.write_allowlist`.
- Generator must never write logs, screenshots, reports, traces, generated eval JSON, or
  temporary manifests outside the harness artifact roots.
- Generator must not use `git add -A`; commits are either disabled or path-scoped.

After generation, call `self_assess()` — catches gaps before the Evaluator via
`submit_assessment` MCP tool. If not confident → extra pass with concerns as `strategic_framing`.

**`self_assess()` always uses Claude** regardless of generator backend. It uses
`cfg.agents.generator.self_assess_model` (default: `claude-haiku-4-5-20251001`).
When generator backend is `codex` or `deepcode`, do not call backend-specific helpers directly
for the extra pass. Call `run_generator` normally — backend dispatch still comes from config.

---

## Phase 4: Evaluator Agent

**Load:** `$SKILL_DIR/instructions/agent-patterns.md` for full implementation.
**Load:** `$SKILL_DIR/instructions/evaluation-rubrics.md` for system prompt + rubric criteria.
**For browser/API/build evidence collection:** load `$SKILL_DIR/instructions/evaluator-evidence.md`.

Two roles: `run_evaluator()` (post-generation gate) + `review_contract()` (pre-sprint criteria review).

```python
# submit_grade schema: contract_results[{id, status, evidence}], rubric_scores{id: 1–5}, feedback
def run_evaluator(spec, contract, app_url, rubric_track="A", cfg=None, project_dir=None) -> EvalResult: ...
```

**⚠️ Deterministic verdict:** Never trust `verdict` from the LLM. Recompute in
`_build_eval_result()` from `contract_results` + `rubric_scores` using `cfg.verdict.*` thresholds.

**Evidence first:** collect deterministic evidence before asking the LLM to grade:
navigation, screenshots, DOM summaries, buttons/forms, viewport overflow, axe, Lighthouse,
API probes, command output, and contract-declared source excerpts. Store all raw evidence under
`harness-state/evidence/sprint-N/`.

---

## Phase 5: Harness Loop

**Load:** `$SKILL_DIR/instructions/iteration-loop.md` for `run_sprint`, `strategic_decision`, `git_commit`.
**Load:** `$SKILL_DIR/instructions/workspace-hygiene.md` before implementing artifact writes.

```python
def main():
    cfg = HarnessConfig.load(Path(__file__).parent / "config.yaml")
    log.setup(PROJECT_DIR, label="run")

def run_sprint(spec, contract, project_dir, handoff=None, cfg=None):
    while iteration < cfg.loop.max_iterations:
        # 1. Generate — try/except; crash is a valid (poor) outcome
        # 2. Self-assess — extra pass if not confident
        # 3. save handoff → harness-state/handoffs/
        # 4. save raw artifacts → harness-state/evidence/sprint-N/
        # 5. git_commit("wip: sprint N iter I") if git checkpointing is enabled
        # 6. Evaluate → EvalResult; save eval → harness-state/evals/
        # 5a. Pass + iteration < min_iterations → quality-improvement continue
        #     Pass + min_iterations met → git_commit("feat") + return
        # 5b. Fail → strategic_decision() → REFINE or PIVOT → set strategic_framing
    # Exhausted: input() if isatty() else return last result
```

Git checkpoints (see `iteration-loop.md` for `git_commit()` helper):

| Event | Message |
|---|---|
| Spec written | `feat: generate spec.md` |
| Contract negotiated | `chore: sprint N contract` |
| Each iteration | `wip: sprint N iteration I` |
| Sprint passes | `feat: sprint N complete` |

For existing-codebase mode, prefer `git checkpoint` or path-scoped staging over `git add -A`.
If path-scoped commit cannot be implemented safely, disable auto-commit and write a diff summary
to `harness-state/evidence/sprint-N/git/`.

Setup: `uv venv && uv pip install claude-agent-sdk pyyaml && export ANTHROPIC_API_KEY=sk-...`
Verify: `uv run python -c "from agents.planner import run_planner; print('OK')"`

---

## Common Mistakes

| Mistake | Fix |
|---|---|
| Trusting LLM's `verdict` field | Recompute in `_build_eval_result()` from `contract_results` + `rubric_scores` |
| Hardcoding model names | Use `cfg.agents.generator.model` — never a string literal |
| Not calling `handoff.save()` before Evaluator | On crash, Evaluator result is lost |
| Letting Generator write in `project_dir` | Set Generator cwd to `project_dir/src`; keep state/logs outside `src` |
| Reusing greenfield boundaries for an existing repo | Switch to existing-codebase mode and define allowlists/protected paths |
| Scattering screenshots, build logs, reports, and JSON through app/public/docs | Route all generated artifacts through `harness-state/evidence/` and `harness-logs/` |
| Adding public evidence routes with no cleanup plan | Declare them in the contract and add a cleanup sprint |
| Using `input()` in CI | Guard with `sys.stdin.isatty()` first |
| Accumulating messages across sprints | Each sprint is a fresh `query()` call — no cross-sprint history |
| Marking `completed_features` from Generator claim | Only promote after Evaluator PASS verdict |
| Using Codex evaluator for UI sprints | Codex can't open a browser — use Claude or deepcode evaluator for Playwright testing |
| Calling `self_assess()` with Codex model kwargs | `self_assess` always uses Claude SDK; model comes from `self_assess_model` |
| Setting `temperature` when `thinking.enabled: true` | Omit temperature or set to `1.0`; the API enforces this |
| Using `codex` backend for Planner expecting interactive Q&A | Codex Planner is single-shot — no clarifying questions loop |
| Treating `deepcode` as raw subprocess | Use Claude Agent SDK with `cli_path=/Users/xzhao/.local/bin/deepcode`, not an arbitrary command adapter |

---

## When to Simplify

| Component | Remove / simplify when |
|---|---|
| Planner agent | User provides SPEC directly |
| Contract negotiation | Human has strong opinions; use config-file mode |
| Generator self-assessment | Evaluator consistently passes first attempt |
| `max_iterations` → 3 | Correctness-only task, no quality/aesthetic goal |
| `min_iterations` → 1 | Early passes are always good enough |
| Refine/pivot `strategic_decision` | Single sprint or correctness task |
| `HandoffState` | Sprint fits in one context window |
| Evaluator | Task within Generator's reliable baseline |