ai-research-harness

cognitive discipline for AI-native scientific experimentation. guardrails, not recipes.

intent

use this skill when you're setting up a new AI-native research experiment repo, designing controlled experiments with LLM agents, building reproducible evaluation pipelines, or structuring a research workspace for long-running agent collaboration. the skill teaches agents how to reason about experiments, not which commands to run. trigger phrases: "research harness", "experiment framework", "controlled experiment", "reproducibility", "evaluation pipeline", "agent collaboration research", "effect size", "claim boundary", "calibration gate".

this skill doesn't prescribe what experiment to run. it prescribes how to think while running it. research agents fail because they scale before validating the minimum loop, overclaim what data proves, treat surprising results as methodology failures before checking execution, delete failed runs to hide progress gaps, change baselines silently, or invent designs without tracing them to published precedents. the antidote is cognitive discipline enforced by repo structure, not prompt reminders.

inputs

repo structure:

• existing or new experiment directory where you'll scaffold three-layer architecture (thinking-space, execution-layer, code-workshop)
• git access (local or remote) for version control on protected surfaces

external connections:

• model API access (OpenAI, Anthropic, or local LLM) with valid credentials in env vars (OPENAI_API_KEY, ANTHROPIC_API_KEY, etc.). gate all model calls behind dry-run validation before smoke run.
• optional: experiment tracking platform (Weights & Biases, MLflow) for run metadata and cross-run comparison. env var: WANDB_API_KEY or MLFLOW_TRACKING_URI.
• optional: data warehouse or artifact store (S3, GCS, local filesystem) for storing raw results and manifest. env var: ARTIFACT_BUCKET or ARTIFACT_PATH.

human context:

• research direction statement (1-3 sentences on what question you're answering)
• baseline model or system to compare against
• preliminary hypothesis or design sketch (can be rough)
• list of metrics or evaluation criteria you care about

reference documentation (provided or authored):

• references/repo-architecture.md (three-layer repo layout, module contracts, manifest schema, validators)
• references/experiment-design.md (progressive building, controlled groups, gold checklists, metric fairness)
• references/scoring-statistics.md (dual-track validation, effect size, reproducibility flags)
• references/scientific-thinking.md (six cognitive disciplines, why they matter)
• references/agent-collaboration.md (governance rules, evidence status, alignment doc template)
• references/methodology-grounding.md (design justification template, grounding framework)

procedure

phase 0: scaffold

step 1 (input: research direction statement, baseline model name, preliminary hypothesis) create root directory structure. folders: thinking-space/, execution-layer/, code-workshop/, harness/, experiments/results/, references/, sync/upstream_proposals/.

output: empty directories with .gitkeep files. commit message: "init: research harness scaffold".

step 2 (input: research direction, metrics list, human owner name) author root entry files:

• AGENTS.md: workspace map (folder purpose, human owner, agent roles, entry points for each phase).
• PLAN.md: phase panel (current phase, phase-gate criteria, next milestones, blockers).
• WORKFLOW.md: procedure (copy of numbered steps from the skill, adapted to your experiment).
• harness/README.md: governance rules (reference the seven governance rules, declare which are active, mark any exceptions with justification).

output: four markdown files. each must be valid markdown. commit message: "docs: root entry files for phase 0".

step 3 (input: existing references/ docs or willingness to author them) ensure all six reference documents exist in references/. if any are missing, create stub sections (headers + placeholder text) so validators won't fail on missing files.

output: six markdown files in references/ (can be stubs). commit message: "docs: reference stubs (fill incrementally)".

step 4 (input: none, automated) run scripts/validate_repo_state.py (mode: WARN). check output for schema violations, missing files, or permission issues. if mode is WARN, non-critical issues print but don't block. if mode is STRICT, any FAIL blocks phase 1 entry.

output: validation log (stdout or file). capture any WARN or FAIL messages. commit message: "chore: phase 0 validation gate (WARN mode)".

decision point: if validation reports FAIL, fix reported issues and re-run before moving to phase 1. if validation reports only WARN, document the warnings in PLAN.md and decide whether to fix before or after phase 1.

phase 1: harden

step 5 (input: list of core modules you'll need, e.g., "task-loader", "model-caller", "scorer", "analyzer") for each module, author code-workshop/<module>/CONTRACT.md. template: purpose (one sentence), inputs (parameter names, types, examples), outputs (return type, example shape), invariants (preconditions, postconditions), local validator (name of script that checks this module's outputs).

output: one CONTRACT.md per module. each must declare a validator script name. commit message: "docs: module contracts for ".

step 6 (input: each module's CONTRACT.md) author scripts/validate_<module>.py for each module. schema: accept input artifact (JSON or CSV), check that outputs match contract shape and invariants, report PASS or FAIL per row/record. aggregate script: scripts/validate_repo_state.py calls all validate_* scripts and reports layer-level status (schema-valid, evidence-governed, planning-useful, experiment-ready).

output: Python scripts. each must run without unhandled exceptions on sample data. commit message: "scripts: validators for ".

step 7 (input: none, automated) create experiments/results/manifest.csv with schema: run_id, wave, task_id, group, model, version_metadata (as JSON blob), status (queued/running/complete/failed), retry_of, git_commit, timestamp, cost_tokens, latency_seconds, error_msg.

output: CSV file with header row and zero data rows. commit message: "chore: experiment manifest schema initialized".

step 8 (input: phase 0 and phase 1 artifacts) run full validation in WARN mode: scripts/validate_repo_state.py --mode=warn. fix any FAIL or CRITICAL WARN. once zero FAIL, set mode to STRICT in config file for future runs.

output: validation log showing all-green status. commit message: "chore: phase 1 validation gate (STRICT mode enabled)".

decision point: if validators report FAIL on protected surfaces (baselines, rubrics, raw results), you have entered a protected-surface change flow (step 11 below). if validators report FAIL on module contracts, fix the contract or the module, not the validator.

phase 2: design

step 9 (input: research question, preliminary hypothesis, baseline model, list of candidate treatments) progressively build experiment design:

• author execution-layer/experiment_design.yaml with: experiment_id, research_question, hypothesis, baseline_config, treatment_configs (each treatment is one YAML dict specifying model, prompt template, or system change).
• for each config, ensure adjacent configs differ by exactly one variable. document the variable in notes: field.

output: YAML file with ≥2 groups (baseline + ≥1 treatment). validate against references/experiment-design.md guidelines. commit message: "docs: experiment design schema v1".

step 10 (input: experiment_design.yaml, preliminary task set) author gold checklists. for each task, create planning_gold (items all groups can achieve) and evidence_gold (items only treatment groups can access). also declare must_include and forbidden for each task.

author references/scoring-statistics.md section on metric fairness: for each metric, declare evidence_access_required (true/false). separate scores by access level in final report.

output: gold checklist YAML and metric fairness annotation in scoring doc. commit message: "docs: gold checklists and metric fairness declarations".

step 11 (input: none, governance-driven) if you need to change any protected surface (baseline, rubric, raw results, scoring config, schema), follow this flow:

• create proposal file: sync/upstream_proposals/proposal_<date>_<change-type>.md.
• proposal contents: what surface, what change, why (rationale, impact on existing results), version bump (e.g., baseline_v1 to baseline_v2).
• submit for human review before applying.
• once approved, apply change, increment version, and re-run validators in STRICT mode.
• log change in harness/CHANGELOG.md.

output: proposal file + approval record + version bump. commit message: "proposal: (approved)".

step 12 (input: experiment_design.yaml with ≥2 groups, research_question, hypothesis) for any non-trivial schema, scoring system, or experiment protocol (new = no direct published precedent), author a Design Justification Document: references/methodology_justification_<design-name>.md. template: problem statement, proposed design, ≥2 published precedents that inspired it (citation + doi + relevant excerpt), explicit assumptions (if no precedent found), and rationale linking design to hypothesis.

output: markdown file citing ≥2 sources or listing explicit assumptions. human review required before proceeding to phase 3. commit message: "docs: design justification for ".

decision point: if design references published precedent and assumptions are explicitly stated, proceed. if design is marked "novel" and no justification exists, return to this step before executing.

step 13 (input: experiment_design.yaml, gold checklists, module contracts) create minimal task set: 1-3 representative tasks. validate task YAML against schema. output: experiments/tasks/v1/task_001.yaml ... task_003.yaml. run scripts/validate_tasks.py and confirm zero FAIL. commit message: "tasks: minimal task set v1 (validated)".

step 14 (input: task set, experiment_design.yaml, module contracts) run dry run: print prompt for each task in each group, no model API call. output: experiments/logs/dry_run_<timestamp>.log. confirm prompts are well-formed and group-specific variables are substituted correctly. commit message: "logs: dry run completed (no api calls)".

step 15 (input: dry run output, model API credentials, smoke-run config) run smoke run: 1 task × 2 groups × 1 retry, hit model API, verify output parsing and manifest entry. expect 2 rows in manifest. output: experiments/logs/smoke_run_<timestamp>.log, manifest updates, raw result files in experiments/results/raw/smoke_<timestamp>/.

decision point: if output parsing fails, fix schema contract and module validator before advancing. if model API call fails (auth, rate limit, timeout), check credentials, retry with backoff (2s, 5s, 10s max 3 times), then escalate to human if still failing.

commit message: "logs: smoke run completed (2 results)".

phase 3: execute & analyze

step 16 (input: smoke run successful, task set, treatment configs, cost budget) run wave 1: small set × all groups (e.g., 10 tasks × 3 groups = 30 calls). batch calls with exponential backoff on rate-limit (429) errors. write each result to manifest immediately after call completes. output: manifest rows, raw results in experiments/results/raw/wave1_<timestamp>/.

log any errors (hallucination, parse failure, timeout, cost overrun). include retry logic: if a single call times out or fails, mark as retry_of: <original_run_id> and attempt 1 retry. if 2nd attempt fails, mark status: failed and log the error.

commit message: "logs: wave 1 execution completed (N results, M failures)".

decision point: if failure rate > 20%, stop and investigate before wave 2. check: model prompt clarity, output schema enforcement, task appropriateness, and cost accounting. fix root cause and re-run the failed tasks.

step 17 (input: wave 1 results, gold checklists, metric fairness annotation) run dual-track scoring. track A (rule-based): parse output for expected fields, check against planning_gold and evidence_gold, award 0/1 points per item. track B (semantic): use a separate model or human evaluator to judge quality, reasoning depth, and specificity. cross-validate: for each result, confirm track A and track B agree on pass/fail. if they disagree, log as a conflict and mark for manual review.

output: experiments/results/scores/wave1_scores.csv with columns: run_id, task_id, group, track_a_score, track_b_score, conflict (bool), notes. separate track A and track B aggregate scores; never mix them into a single total.

commit message: "scores: wave 1 dual-track scoring completed".

step 18 (input: wave 1 scores, both tracks) compute statistics. primary metric: Cohen's d (effect size, group vs. baseline). secondary: 95% confidence interval (via bootstrap or t-test), paired t-test (if results are paired), Wilcoxon rank-sum (if non-normal). output: experiments/results/statistics/wave1_stats.json with: metric, value, ci_low, ci_high, p_value, sample_size, notes.

add reproducibility flag: --reproduce script that re-runs statistics on same data, confirming bit-for-bit match. commit message: "stats: wave 1 statistics (reproduce-able)".

step 19 (input: wave 1 results, error logs, failed task list) analyze errors by category: hallucination (output contradicts input facts), depth (incomplete reasoning), specificity (vague or generic output), task-inappropriateness (task unclear or unsolvable). for each category, count frequency and cite examples. output: experiments/results/analysis/wave1_errors.md with sections per category, examples, and hypothesis (why did this error occur).

commit message: "analysis: wave 1 error analysis completed".

phase 4: handoff & writing

step 20 (input: wave 1 scores, effect sizes, error analysis) write claim-safe memo: separate supported findings (strong signal + low error rate + effect size > Cohen's d = 0.2), positive signals (trend, but not yet strong), not-yet-supported claims (intriguing but needs more data), and required next evidence (what data would prove/disprove).

output: thinking-space/claim_safe_memo_wave1.md (1-2 pages, human readable). human review required before any expansion claim. commit message: "memo: claim-safe findings from wave 1".

decision point: if wave 1 shows strong signals, do not immediately expand to wave 2. instead, proceed to step 21 (calibration gate). if wave 1 shows no signal or negative signal, either pivot hypothesis, adjust design (return to step 9), or halt and document null result.

step 21 (input: strong signals from wave 1, manifest, raw results, scores, error analysis) execute calibration gate before scaling: lock down claim boundaries (finalize what you're actually claiming), audit artifacts (re-validate all results for data integrity, no overwrites, no deletions), fix provenance gaps (confirm git commit hashes, timestamps, and model versions are recorded for every result).

output: thinking-space/calibration_gate_wave1.md (checklist: claim boundaries locked, artifacts audited, provenance complete, protected surfaces frozen). human sign-off required. commit message: "gate: calibration checkpoint completed".

step 22 (input: calibration gate sign-off, alignment doc template) author alignment doc: execution-layer/HANDOFF_WAVE1_TO_WAVE2.md (~1 page). sections: current state (wave 1 results summary), entry files (files the next agent should read first), new surfaces (new protected surfaces added in phase), preflight commands (run these before starting wave 2), protected surfaces (list of locked files, version info).

never include chat history or long context. output: single markdown file. commit message: "docs: handoff alignment for wave 2".

step 23 (input: wave 1 results, memo, alignment doc, statistics) if expanding to wave 2: return to step 16 with larger task set (e.g., 50 tasks × all groups). repeat steps 16-20 for wave 2. compare wave 2 statistics to wave 1; if effect size grows or error rate drops, signal strengthens. if effect size shrinks or error rate rises, investigate design drift (did something change between waves?).

output: wave 2 logs, scores, statistics, error analysis, new claim-safe memo.

step 24 (input: final results, all statistics, error analysis, external papers) write research document (paper, report, blog): sections are findings (supported + strong signal), methodology (design justification doc), results (effect size + CI + examples), limitations (what didn't work, null results, confounds). mark any AI-generated numbers with [NEEDS-VERIFICATION]. include full experiment trace (manifest, git commits, dates, model versions, costs).

output: markdown or LaTeX file. human review + fact-check required before publication. commit message: "draft: research report (needs review)".

decision points

credential & connectivity failures (any phase)

• if model API call fails due to auth: check env var, credentials format, and API key expiry. retry once. if still failing, report error to human.
• if data warehouse upload fails (rate limit, quota, network timeout): retry with exponential backoff (2s, 5s, 10s). if 3 retries exhaust, log error and cache locally, sync later.
• if git push fails (network, permission): confirm remote, credentials, and branch permissions.