Autoresearch Agent

you sleep. the agent experiments. you wake up to results.

autonomous experiment loop inspired by Karpathy's autoresearch. the agent edits one file, runs a fixed evaluation, keeps improvements, discards failures, and loops indefinitely.

not one guess , fifty measured attempts, compounding.

intent

use this skill when you want to optimize any file (code, prompts, content, config) by a measurable metric without manual guessing. the agent autonomously edits the target file, evaluates changes against a baseline, commits improvements via git, reverts failures, and repeats. works for code speed, bundle size, test pass rate, prompt quality, content CTR, or any domain with a quantifiable metric and an evaluation command. activate this when you say "make this faster," "optimize [file] for [metric]," "improve my headlines," or "run experiments overnight." requires a git repo, a target file, and an evaluation command that outputs a metric value.

inputs

required

• target_file (string, filepath): the file the agent will edit. must be tracked in git. examples: src/api/search.py, content/titles.md, prompts/system.txt.
• evaluate_cmd (string, shell command): the command that measures success. must output a metric value in format metric_name: value to stdout. examples: pytest bench.py --tb=no -q, python evaluate.py, npm run build --stats.
• metric (string): the name of the metric to extract from eval output. examples: p50_ms, size_bytes, pass_rate, ctr_score.
• metric_direction (enum: "lower" or "higher"): whether the metric improves by going down (lower) or up (higher). examples: lower for latency/size, higher for pass rate/quality score.
• git_repo (filepath): root of a git repository. the agent will create branches, commit, and reset within this repo.

optional

• program_md (string, filepath or text): human-readable objectives, constraints, and strategy. if file exists at .autoresearch/{domain}/{name}/program.md, the agent reads it. if not provided, the agent asks for clarity on what can/cannot change.
• time_budget_minutes (integer, default 5): max seconds allowed per evaluation run. if eval exceeds 2.5x this budget, kill it and log as crash.
• evaluator_type (enum: "benchmark_speed", "benchmark_size", "test_pass_rate", "build_speed", "memory_usage", "llm_judge_content", "llm_judge_prompt", "llm_judge_copy", "custom"): which evaluation template to use. if custom, user provides their own evaluate.py.
• scope (enum: "project" or "user", default "project"): where to store experiment config and results. "project" stores .autoresearch/ in repo root (git-tracked, shareable). "user" stores in ~/.autoresearch/ (personal, isolated).
• domain (string, default "custom"): category for organizing experiments. examples: engineering, marketing, content, prompts, custom.
• experiment_name (string): human-readable name for this experiment. examples: api-speed, medium-ctr, bundle-shrink.

external connections

• git_cli (required): local git binary (usually present). the agent runs git checkout, git add, git commit, git reset --hard.
• python_runtime (required): python 3.8+. the agent runs evaluation scripts and helper utilities.
• llm_subscription (conditional, if using llm_judge_* evaluators): existing CLI tool subscription (Claude, Codex, Gemini) that user already runs. evaluation calls the same tool, no additional cost beyond user's current plan. requires cli tool configured with api key in env var (ANTHROPIC_API_KEY for Claude, OPENAI_API_KEY for Codex, GOOGLE_API_KEY for Gemini).

edge cases and validation

• target_file not in git: agent will fail on first commit. pre-requisite: run git init && git add . && git commit -m 'initial' if repo is new.
• evaluate_cmd fails on baseline: evaluation will crash. agent validates this before starting loop. proactive trigger: test eval command manually once before starting.
• metric not found in eval output: agent logs "N/A" and treats run as crash if metric is missing.
• network timeout on eval: if eval command makes external requests (api calls, llm judgments) and times out, treated as crash after time_budget_minutes × 2.5.
• llm subscription expired or rate-limited: llm_judge_* evaluators will fail. agent logs crash. manual fix: refresh api key in env var or upgrade subscription.
• git branch conflict: if experiment branch already exists and has diverged from main, agent warns before proceeding.

procedure

phase 1: setup (one time)

1. input: user provides target_file, evaluate_cmd, metric, metric_direction, domain, experiment_name, scope, optional program_md.
2. validate evaluate_cmd: run it once on baseline target_file in a sandbox. capture stdout. check metric value is parseable. if eval fails or metric not found, halt and ask user to fix evaluate_cmd.
3. create directory structure:
   • if scope == "project", create .autoresearch/{domain}/{experiment_name}/ in git repo root.
   • if scope == "user", create ~/.autoresearch/{domain}/{experiment_name}/.
   • create subdirs: none needed, all files flat.
4. write config.cfg: save target, evaluate_cmd, metric, metric_direction, time_budget_minutes as key-value pairs.
5. write or read program.md: if user provided program_md content or file path, use it. else create template with sections: objectives, constraints, strategy, notes. ask user to fill in objectives and constraints if not provided.
6. write results.tsv header: columns: commit | metric | status | description. no rows yet.
7. copy evaluator script (if --evaluator flag used): copy ready-made evaluate.py (benchmark_speed, llm_judge_content, etc.) into experiment dir. mark as read-only (chmod 444 on unix).
8. create git branch: git checkout -b autoresearch/{domain}/{name}. commit empty results.tsv and config files to this branch.
9. output: print setup complete, directory path, first iteration instructions.

phase 2: experiment loop (agent runs repeatedly, each call is one iteration)

1. input for each iteration: experiment name (domain/name), user's implicit request to run once (--single flag) or continuous loop (--loop flag).

2. load config and history:

   • read .autoresearch/{domain}/{name}/config.cfg into memory (target, evaluate_cmd, metric, metric_direction, time_budget).
   • read .autoresearch/{domain}/{name}/program.md for strategy and constraints.
   • read .autoresearch/{domain}/{name}/results.tsv into memory. parse rows as list of (commit_hash, metric_value, status, description).

3. ensure on correct branch: git checkout autoresearch/{domain}/{name}. verify HEAD matches latest commit hash in results.tsv (if results.tsv is non-empty).

4. decide next change:

   • review results.tsv: which changes kept? which crashed? which directions (code style, algorithm, parameter) haven't been tried?
   • apply strategy escalation: runs 1-5 are obvious wins (caching, removing debug prints), runs 6-15 are systematic (vary one hyperparameter at a time), runs 16-30 are structural (swap algorithm), 30+ are radical (rewrite subsystem).
   • check simplicity criterion: prefer small improvement with simple code over big improvement with complex code.
   • decide one atomic change to target_file (one variable, one function, one config line).
   • output to user: what you're about to try and why.

5. edit target_file: make the single change. do not modify evaluate.py or results.tsv.

6. commit change: git add {target_file} && git commit -m "experiment: {description of change}". capture commit hash.

7. run evaluation:

   • execute evaluate_cmd in a subprocess with timeout of time_budget_minutes * 2.5 seconds.
   • capture stdout and stderr.
   • if subprocess exceeds timeout, kill it, log "crash (timeout)" in status, log metric as "N/A", skip to step 10.
   • if subprocess returns non-zero exit code, log "crash ({error type})" in status, log metric as "N/A", skip to step 10.
   • if subprocess succeeds, parse stdout for metric: value pattern. extract value as float. if metric not found in output, log "crash (metric not found)", log metric as "N/A", skip to step 10.

8. compare to baseline (if results.tsv has any previous "keep" rows):

   • find best previous metric value among rows where status == "keep".
   • if metric_direction == "lower", new_value < best_value is improvement. if metric_direction == "higher", new_value > best_value is improvement.
   • if improvement (or first run with no previous keeps), set status to "keep". else set status to "discard".

9. handle status:

   • if status == "keep": leave commit in place. log to results.tsv.
   • if status == "discard": git reset --hard HEAD~1. log to results.tsv (this row records the discarded attempt, metric, and description for learning).
   • if status == "crash": git reset --hard HEAD~1. log to results.tsv. count consecutive crashes (crashes in last 5 iterations). if 5+ consecutive crashes, halt and alert user with message "5 consecutive crashes detected. pause and review strategy in program.md or try different approach."

10. log result: append row to results.tsv: {commit_hash} | {metric_value} | {status} | {description}.

11. self-improvement check: if total rows in results.tsv is multiple of 10 (i.e., every 10 experiments), scan results.tsv for patterns. examples: "all cache-related changes keep", "refactoring attempts never help metric", "simple parameter tweaks most reliable". append findings to program.md under "Learned Patterns" section. this informs future iterations.

12. decide next action:

• if --single flag: stop here, return results.tsv row and summary to user.
• if --loop flag: sleep for configurable interval (10m, 1h, daily, weekly, monthly). go to step 2.
• if user interrupted: halt, ensure results.tsv saved, commit any pending changes.

phase 3: viewing and exporting results

1. input: user requests log_results.py with flags like --experiment, --domain, --dashboard, --format, --output.
2. read results.tsv for specified experiment(s).
3. parse and aggregate: for each experiment, compute: run count, kept count, best metric, delta from start, status (active/paused/done).
4. format output:
   • if --format tsv: output raw results.tsv.
   • if --format csv: convert to csv, proper quoting.
   • if --format markdown: render as markdown table with headers.
   • if --dashboard: show summary table across all experiments or domain.
5. output to stdout or file (if --output specified).

decision points

• if evaluate_cmd fails on baseline: halt setup. output error, ask user to debug eval command offline and retry setup.

• if metric_direction not specified: ask explicitly: "is lower or higher better for {metric}?" no default. must have answer before loop starts.

• if target_file not in git: halt before first commit. output: "target_file must be in a git repo. run git init && git add . && git commit -m 'initial' first."

• if results.tsv shows no improvement in last 20 runs: suggest (do not force) reviewing and updating strategy section of program.md. offer to continue or pivot to new approach.

• if agent attempts to modify evaluate.py: hard stop. output: "evaluate.py is locked. modifying it invalidates all comparisons. revert and try different target_file change." do not commit the change.

• if 5 consecutive crashes occur: halt loop. alert user: "5 consecutive crashes. likely fundamental issue with target or eval setup. pause, debug, resume when ready." provide recent crash descriptions to help diagnose.

• if eval takes longer than time_budget_minutes: log timeout crash on first few runs. proactive: on setup, measure eval time on baseline. if baseline eval is slower than time_budget, suggest increasing time_budget.

• if llm_judge_ evaluator called but api key missing or expired*: eval crashes with auth error. agent logs crash. alert user to refresh api key in env var.

• if experiment branch conflicts with existing branch: warn user: "branch autoresearch/{domain}/{name} already exists and may have diverged. reset to main and restart, or resume existing experiment?" user chooses.

• if results.tsv is corrupted or unreadable: agent falls back to empty history, logs warning, continues from run 1.

output contract

setup phase outputs

• directory structure created at .autoresearch/{domain}/{name}/ (project scope) or ~/.autoresearch/{domain}/{name}/ (user scope).
• config.cfg file contains keys: target, evaluate_cmd, metric, metric_direction, time_budget_minutes. format: ini-style or yaml.
• program.md file contains human-readable sections: objectives, constraints, strategy, learned patterns (updated every 10 runs).
• results.tsv file created with header row commit | metric | status | description. initially empty (header only).
• evaluate.py (if applicable) copied into experiment dir, marked read-only.
• git branch autoresearch/{domain}/{name} created and checked out. initial commit recorded.

per-iteration outputs

• results.tsv appended with one row per iteration: {git_commit_hash} | {metric_value_or_NA} | {keep|discard|crash} | {human_description}.
• git log shows one commit per kept experiment (discarded and crashed runs do not appear in log, only in results.tsv).
• stdout from agent includes: change attempted, eval output snippet, status (keep/discard/crash), metric value, next strategy.

viewing outputs

• single experiment view: tabular display of all runs, columns: iteration#, commit, metric, status, description. sortable by metric or date.
• dashboard view: summary table across experiments, columns: domain, name, run_count, kept_count, best_metric, delta_from_start, status.
• export formats: tsv (raw), csv (quoted), markdown (github-compatible table).

outcome signal

success criteria for user to confirm skill worked:

1. setup succeeded: .autoresearch/{domain}/{name}/ directory exists, config.cfg and program.md are readable, results.tsv header is present, no errors during setup.

2. first iteration ran: results.tsv has at least one data row (not just header). status is "keep", "discard", or "crash", metric_value is numeric or "N/A".

3. improvements compound: after 10+ iterations, best_metric has moved in the metric_direction (lower for latency, higher for quality). ideally 5-10% improvement per 10 runs.

4. git history is clean: git log autoresearch/{domain}/{name} shows only kept commits. discarded and crashed runs do not pollute history.

5. loop continues autonomously: if --loop flag active, agent wakes every interval (10m, 1h, daily, etc.) and runs new iteration without user intervention.

6. self-improvement visible: after 20+ iterations, program.md "Learned Patterns" section contains actionable insights (e.g., "parameter X always helps", "avoid structural changes, simple tweaks work better").

7. no manual intervention needed: agent handles crashes, reverts, and logging without user restarting or debugging each run.

8. results exportable and readable: python scripts/log_results.py --experiment {domain}/{name} --format markdown produces human-friendly table. dashboard view shows progress across multiple experiments.

9. resumable state: if process stops (context limit, user pause), next session resumes from results.tsv state. no runs are lost or repeated.

related skills

• self-improving-agent: improves an agent's own memory and rules over time. not for structured experiment loops. complementary if autoresearch agent needs to refine its own strategy mid-loop.
• senior-ml-engineer: ml architecture decisions and design review. complementary for initial system design, autoresearch optimizes after design is stable.
• tdd-guide: test-driven development. complementary, test suite can serve as evaluation_cmd (metric: pass_rate).
• skill-security-auditor: audit skills before publishing. orthogonal, not for optimization loops.

credits: original author Alireza Rezvani, inspired by Karpathy's autoresearch. enriched for Implexa quality standards.