Structured Multi-Agent Deliberation

SKILL.md

---
name: structured-deliberation
description: Provides a structured multi-agent deliberation framework with role schemas (action/guardian/observer/critic), verification protocols, and stopping criteria. Activate when designing multi-agent systems for non-trivial deliberation, instrumenting agent debates against sycophancy, or seeking an evaluable alternative to free-form group prompting.
version: 0.1.0
---

# Structured Multi-Agent Deliberation Framework

A protocol for running multi-agent deliberation that produces *evaluable* output — claims with explicit lifecycle, verifications with cross-agent evidence, decisions that cite specific support, and stopping criteria that prevent both premature consensus and infinite loops.

The central premise: **free-form multi-agent prompting produces consensus that looks robust but isn't.** Without structural cross-validation, agents drift toward agreement (sycophancy cascade); without explicit claims, decisions can't be traced to evidence; without stopping criteria, deliberations either stop too early (premature consensus) or run forever (artifact accumulation).

This framework provides:

1. **4 role schemas** (Action / Guardian / Observer / Critic) — each contributing a perspective the others can't substitute
2. **4 cross-validation checks** per round — artifacts are not siloed; they must contact each other
3. **Claims + verifications infrastructure** — disagreements become testable claims with status lifecycles
4. **6 goal-driven stopping criteria** — explicit signals for when the deliberation should end
5. **Stress test protocol** — forced agent absence reveals which roles are load-bearing

## When to use

Activate this skill when:

- Designing a multi-agent system for non-trivial deliberation (not chitchat or task delegation, but actual disagreement-resolving discussion)
- Instrumenting an agent debate to detect sycophancy / convergence quality
- Seeking an evaluable alternative to free-form group prompting (e.g., "5 LLMs discuss X")
- Running architectural / strategic / methodological reviews where cross-perspective rigor matters
- Building a multi-agent system that needs `claims.jsonl` / `verifications.jsonl` / `decisions.jsonl` audit trails

Don't activate when:

- The user wants simple multi-agent task delegation (not deliberation)
- A single LLM with chain-of-thought is sufficient (no real perspective diversity needed)
- The user is asking about LLM ensemble methods for accuracy gains (different problem)

## The 4-role structure

| Role | Question | Artifact |
|---|---|---|
| **Action** | "Where's the verb?" | Behavioral Trace Set |
| **Guardian** | "What's the invariant?" | Identity Invariant Check |
| **Observer** | "Can a real operator navigate this?" | Operator Walkthrough |
| **Critic** | "What's the strongest argument against this?" | Adversarial Test Case |

Full role definitions: [references/role-schemas.md](references/role-schemas.md).

Why these 4: each contributes a perspective the others *can't substitute*. Action grounds in state transitions; Guardian formalizes invariants; Observer tests usability; Critic forces adversarial scrutiny. Without one, the deliberation has a structural blind spot.

## Round structure (8 phases)

Each round runs through:

```
A. Lead Assignment       → who leads this round
B. Lead Proposal         → 600-800 word position
C. Supplements           → other agents' takes (200-400 words each)
D. Task Phase            → 4 structured artifacts
E. Cross-Validation      → 4 verification checks
F. Claims Update         → new claims, status changes, validate
G. Assessment            → decisions, tensions, architecture impact
H. (every K rounds) Drift Check
```

Full template: [templates/round-template.md.tmpl](templates/round-template.md.tmpl).

## Cross-validation: 4 checks

After artifacts are produced (Phase D), 4 verification checks run before claims update:

1. **Critic attack vs Action trace** — does Critic's attack break Action's traces?
2. **Guardian invariant vs Observer walkthrough** — does Operator path violate the invariant?
3. **Observer friction vs Critic attack** — friendly-fire overlap (most valuable signal)
4. **All artifacts vs prior decisions** — continuity / drift detection

Each check produces a `verifications.jsonl` entry with `evidence_refs` ≥2 and a verdict (BROKEN / ROBUST / UNCLEAR / NOT_APPLICABLE).

Full protocol: [references/verification-protocol.md](references/verification-protocol.md).

## Claims infrastructure

Disagreements become testable claims:

```json
{
  "id": "C-{round}-{sequence}",
  "round": N,
  "raised_by": "action | guardian | observer | critic",
  "text": "the claim itself, 1-3 sentences",
  "testable_as": "single observable outcome",
  "status": "pending | tested_confirmed | tested_refuted | partially_refuted | tested_unclear | superseded"
}
```

Status transitions are driven by verification verdicts. Decisions cite claims and verifications.

Full schema and validator: [references/claims-infrastructure.md](references/claims-infrastructure.md), [scripts/claims-validator.py](scripts/claims-validator.py).

## Stopping: 6 signals

The deliberation should consider stopping when **≥4 of 6 signals** are satisfied:

1. Claim refutation rate stabilizes
2. Disagreement slope flat or rising (no sycophancy)
3. All agents have led at least one non-stress round
4. Stress tests have been executed (≥N-1 agents stress-tested)
5. Drift checks pass
6. Pending claim fraction < 30%

Full criteria + detector script: [references/stopping-criteria.md](references/stopping-criteria.md), [scripts/stopping-detector.py](scripts/stopping-detector.py).

## Stress tests

Forced agent absence reveals load-bearing-ness. If a stress round looks identical to a normal round, the absent agent's role is decorative.

Recommended schedule for a 13-round deliberation:

- R5: STRESS-CRITIC (test that adversarial pressure is structural)
- R10: STRESS-GUARDIAN (test that invariant work has unique value)
- R12: STRESS-ACTION (test that behavioral grounding has unique value)
- R13: STRESS-OBSERVER (test that operator perspective has unique value)

In stress rounds, the absent agent's artifact is NOT produced. Verifications involving the absent agent's artifact return `NOT_APPLICABLE` with stress reason in notes.

## Failure modes to watch

8 common failure modes with detection signals:

- FM-1: Sycophancy cascade
- FM-2: All-BROKEN performative adversariality
- FM-3: Drift (silent goal-departure)
- FM-4: Single-agent domination
- FM-5: Claim inflation or starvation
- FM-6: Verification bypass
- FM-7: Stress test avoidance
- FM-8: Round budget overrun

Full catalog with detection signals and responses: [references/failure-modes.md](references/failure-modes.md).

## How to use this skill

When triggered, walk the user through 5 stages:

### Stage 1 — Domain elicitation

Ask:

- What system are you building / evaluating?
- What's the deliberation question? (specific, not "let's discuss X")
- What 4 perspectives make sense for your domain? (See role-schemas.md cross-domain examples)
- What's the round budget? (8 minimum, 13 standard)

### Stage 2 — Role configuration

For each of the 4 roles, instantiate for the user's domain:

- Action: who/what produces behavioral traces? (typically: implementation lead)
- Guardian: who/what specifies invariants? (typically: domain expert / safety / compliance)
- Observer: who/what represents the operator? (typically: user-facing role / customer)
- Critic: who/what attacks? (typically: red team / failure-mode adversary)

Use [templates/role-prompt.md.tmpl](templates/role-prompt.md.tmpl) as starting prompts.

### Stage 3 — Round template configuration

Configure:

- Round budget (typically 13: 10 normal + 2 stress + 1 synthesis)
- Stress test schedule
- Drift check interval (typically every 5 rounds)

Use [templates/round-template.md.tmpl](templates/round-template.md.tmpl) as the per-round structure.

### Stage 4 — Run rounds

For each round:

- Run the 8-phase template (A-H)
- After Phase E, run [scripts/claims-validator.py](scripts/claims-validator.py) to verify integrity
- Every 3-4 rounds, run [scripts/stopping-detector.py](scripts/stopping-detector.py) to evaluate stop signals

Use [scripts/round-controller.py](scripts/round-controller.py) as orchestration skeleton (adapt to your LLM API).

### Stage 5 — Synthesis and retrospective

When stop signals satisfied (≥4/6) OR round budget hit:

- Run a synthesis round (no new claims; resolve pending claims; produce summary)
- Write retrospective using the failure-modes catalog as a checklist
- Document any iteration log entries (what calibration revealed)

## Quick example

User: *"I want to evaluate 3 candidate architectures for our agent system, but free-form discussion keeps converging too fast — I think we're sycophanting."*

Stage 1 reveals:

- System: agent architecture comparison (3 candidates)
- Question: which architecture is most robust under load + adversarial conditions?
- Round budget: 13 (standard)

Stage 2 instantiates 4 roles:

- Action: senior backend engineer (produces behavioral traces of each candidate under load)
- Guardian: SRE lead (invariants around uptime / data integrity / observability)
- Observer: customer support lead (walkthroughs of debugging the system as on-call)
- Critic: security lead (adversarial attack hypotheses)

Stage 3 sets:

- 13-round budget; stress tests at R5/R10/R12/R13
- Drift checks at R5 and R10

Stage 4 runs 13 rounds. After R7, sycophancy detector triggers (Signal 2 negative slope). Response: tighten Critic schema, R8 produces a BROKEN verdict on Action's R3 trace.

Stage 5 synthesis: Architecture B emerges as winner not because of consensus, but because:

- Survived 4 stress tests with signature degradation matching predictions (high C2 in MADEF terms)
- Critic attacks on B reproduced as ROBUST 4 of 5 times (vs A: 1/5, C: 2/5)
- Guardian's invariant ("recovery time < 30s under partial outage") only B passed reliably
- Operator walkthroughs found friction at the same place across 3 different operator personas — meaningful signal, not noise

Full walkthrough: [examples/condensed-deliberation.md](examples/condensed-deliberation.md).

## How the skill behaves at each turn

- **Don't** dump the full 4-role schema at the user upfront. Walk them through Stage 1 elicitation first.
- **Don't** start running rounds (Stage 4) before Stage 2 (role configuration) is settled. Roles improperly cast produce noise.
- **Do** push back if the user wants to skip stress tests "to save rounds". Stress tests are the load-bearing-ness check; without them, the deliberation can't be evaluated.
- **Do** run the validator (`claims-validator.py`) after every Phase F. Errors here cascade.
- **Do** trigger sycophancy alarm if Signal 2 (disagreement slope) goes negative for 2+ rounds. Don't wait for the deliberation to end.

## References

- [references/role-schemas.md](references/role-schemas.md) — full 4-role definitions + cross-domain instantiations
- [references/verification-protocol.md](references/verification-protocol.md) — 4 cross-validation checks
- [references/claims-infrastructure.md](references/claims-infrastructure.md) — jsonl schemas
- [references/stopping-criteria.md](references/stopping-criteria.md) — 6 stop signals
- [references/failure-modes.md](references/failure-modes.md) — 8 failure modes catalog

## Templates

- [templates/role-prompt.md.tmpl](templates/role-prompt.md.tmpl) — parameterized 4-role system prompts
- [templates/round-template.md.tmpl](templates/round-template.md.tmpl) — 8-phase round structure
- [templates/artifact-schemas/](templates/artifact-schemas/) — per-role artifact schemas (action / guardian / observer / critic)

## Scripts

- [scripts/claims-validator.py](scripts/claims-validator.py) — JSONL integrity check (run after Phase F each round)
- [scripts/stopping-detector.py](scripts/stopping-detector.py) — evaluate 6 stop signals (run every 3-4 rounds)
- [scripts/round-controller.py](scripts/round-controller.py) — orchestration skeleton (adapt to your LLM API)

## Examples

- [examples/condensed-deliberation.md](examples/condensed-deliberation.md) — 5-round walkthrough on a fictional architecture comparison
- [examples/stress-test-walkthrough.md](examples/stress-test-walkthrough.md) — what stress rounds reveal (R12/R13 patterns)

## What this skill does NOT do

- It does not run the LLM calls for you — `round-controller.py` is a skeleton; you integrate with your LLM API
- It does not validate that *your* 4 roles are domain-correct — that's a calibration question
- It does not work for single-agent deliberation (use chain-of-thought + self-critique instead)
- It does not produce a single ranking score — outputs are claims, verifications, and decisions with provenance, not a scalar

## Source

This skill captures a methodology developed across multiple multi-agent deliberation experiments, frozen as v1 after running on multiple calibration cases. The 4-role structure, 4 cross-validation checks, 6 stopping signals, and stress test patterns all emerged from calibration findings.

Companion skill: [multi-dim-eval-framework](../multi-dim-eval-framework) — for *evaluating* deliberation quality (MADEF). The two skills compose: this skill produces deliberation; that skill evaluates it.

## License

MIT