Skip to main content
skill CX Codex

ml-autoresearch — Claude Code skill: what it does, how to install

ml-autoresearch drives an autonomous machine learning loop where model diagnostics and research papers shape each code iteration.

MIT ★ 185 updated 2026-06-30

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add gaasher/Agent-Loop-Skills
/plugin install agent-loops@agent-loop-skills
Open on GitHub

What ml-autoresearch does

The skill turns Claude Code into an autonomous research agent for machine learning projects. Rather than running blind parameter sweeps, it executes an experiment, inspects run telemetry — gradients, activations, embeddings, and error distributions — and grounds the next hypothesis in that data. Every iteration applies exactly one modification to ensure measurable attribution.

Setup begins by writing loop.run.yaml to specify editable training scripts, run commands, and execution gates based on time or epochs. An optional <literature> switch toggles scientific paper grounding via a companion helper. When enabled, the agent queries papers, scores the evidence, and implements published techniques; when disabled, it runs as a closed-loop diagnostic routine.

Python 3.9+ is strictly required, and the workflow fails if your evaluation pipeline lacks a clear scalar metric. The agent enforces a simplicity rule: a 0.001 gain costing 20 lines of brittle code is rejected, while matching gains achieved by deleting code are accepted.

What it helps with

  • run continuous ML experiments without manual intervention
  • guide model architecture changes using internal gradient and activation metrics
  • ground training modifications in peer-reviewed scientific literature
  • maintain a structured experiment history inside results.tsv

Example prompts for the agent

  • Start an ml-autoresearch loop targeting validation loss in train.py
  • Run the autonomous research loop with literature set to on
  • Set up an ml-autoresearch sandbox with a 15 minute budget per run

How to install ml-autoresearch

1Plugin marketplace

/plugin marketplace add gaasher/Agent-Loop-Skills
/plugin install agent-loops@agent-loop-skills

This installs the whole agent-loops plugin with all its skills

2Skills CLI (Vercel)

npx skills add gaasher/Agent-Loop-Skills --skill ml-autoresearch -a claude-code

3Manually: put the files in place

~/.claude/skills/ml-autoresearch/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/ml-autoresearch/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about ml-autoresearch

How to install ml-autoresearch in Claude Code?

In Claude Code, run /plugin marketplace add gaasher/Agent-Loop-Skills followed by /plugin install agent-loops@agent-loop-skills. Alternatively, use npx skills add gaasher/Agent-Loop-Skills --skill ml-autoresearch -a claude-code.

What does ml-autoresearch do?

It coordinates an ongoing training loop that studies internal model diagnostics after each run, modifies code within permitted files, and tracks results in an experiment ledger.

When should the literature option be enabled?

Keep it off for pure self-contained diagnostic runs, or switch it to on when working on established problems where extracting methods from scientific papers can guide the next step.

Source text

SKILL.md · MIT
---
name: ml-autoresearch
description: >
  Use when the user wants an autonomous ML research loop that does more than blindly try changes.
  After every training run the agent analyses what actually happened inside the model — gradients,
  activations, embeddings, errors, data — and grounds the next change in that evidence. A `<literature>`
  on/off dial adds scientific-literature grounding: off behaves as a pure analysis-first loop; on
  searches papers, grades the evidence, and implements only what prior work supports. One change per
  run; loops forever until interrupted. Not for one-off training runs or hyperparameter sweeps.
compatibility: Requires Python 3.9+
metadata:
  version: "0.1.0"
---

# ML Autoresearch Loop

This loop is **analysis-first**: every experiment is followed by a diagnostic pass that examines what
happened inside the model, and the next change is a hypothesis grounded in that evidence — not a guess.
The feedback signal is `<metric>` read from the run log; the analysis is the spine that decides what to
change. A `<literature>` dial (`on`/`off`) optionally grounds each change in prior work via the sibling
`literature-search` skill. One change per iteration, so each metric move is attributable.

You are the researcher. Do not pause to ask for permission once the loop is running.

## When to use
Use for an open-ended, autonomous ML research campaign where you want each change motivated by analysis
of the model's actual behaviour. Set `<literature> = off` for a self-contained analysis-and-score loop;
set `<literature> = on` to additionally ground changes in the scientific literature (paper search,
evidence grading, a reusable findings backlog). Not for a single training run, a fixed sweep, or tasks
with no measurable scalar metric. Default to `off` unless the user wants literature grounding or the
problem is a known, well-published one where prior recipes will pay off.

## Setup
**Resolve bindings interactively.** If `loop.run.yaml` exists in the working dir, load it and skip to
the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is available — record `<host>` =
`claude-code`) infer a likely value for each binding from the project and present it as the recommended
option; on other hosts (`<host>` = `other`) ask each as a quoted plain-text prompt. Then write
`loop.run.yaml` (format: `examples/run.example.yaml`) and **confirm every value with the user before
creating any other files.** For `branches` strategy, create `git checkout -b autoresearch/<run_tag>`
(tag from today's date; branch must not exist). For `time` gating, write `<sandbox_root>/run_with_timeout.sh`
(`timeout $(( <budget> * 60 )) <entrypoint> "$@"`) and use it as the run command, hard-killing at
`2 × <budget>` min; for `epochs`, patch the epoch cap in an `<editable_files>` file.

| binding | meaning | default | how to infer |
|---|---|---|---|
| `<metric>` / `<metric_direction>` | scalar to optimize + `minimize`/`maximize` | — | scan editable files + README for metric names |
| `<run_cmd>` / `<entrypoint>` | command that runs one experiment end to end | — | `pyproject.toml` / `.venv` / README |
| `<editable_files>` | files fair game to edit (never the eval harness) | — | model / config / train scripts; exclude data, logs, env, harness |
| `<sandbox_root>` | where snapshots + ledgers live | `./sandbox` | next to the editable files |
| `<iter_strategy>` | `snapshots` or `branches` | `snapshots` | is the working dir a clean git repo? |
| `<gate>` / `<budget>` | `time` (min) or `epochs`, and the limit | — | existing time/epoch settings in config |
| `<literature>` | `on` = literature-grounded; `off` = analysis-only | `off` | does the user want paper grounding? |
| `<research_scale>` *(on only)* | depth dial `low`/`medium`/`high`/`x-high` | `medium` | see roles/research-subagent.md |
| `<domain>` *(on only)* | one-phrase problem domain; seeds query phrasing only, never filters | — | infer from data/model/task |
| `<lit_skill_dir>` *(on only)* | install dir of the `literature-search` skill | sibling of this loop | `~/.claude/skills/literature-search/` (adjust per host) |
| `<lit_py>` *(on only)* | Python ≥3.9 interpreter for the lit helper (stdlib-only) | `python3` | independent of `<run_cmd>` |

**FILE EDIT GUARD**: before touching any file at any point — setup or loop — confirm it is in
`<editable_files>`, because everything else is read-only ground truth (the eval harness defines
`<metric>`). No exceptions.

### Literature toolchain (only when `<literature> = on`)
Paper search goes through the sibling **`literature-search` skill** (stdlib-only, no installs):
`<lit> = <lit_skill_dir>/tools/lit_search.py` (note the `tools/` segment). Reuse one cache by
appending `--cache-dir <sandbox_root>/literature/.cache` after the subcommand. Subcommands print JSON;
on failure they print `{"error","fallback"}` and exit non-zero — then **degrade to the host's
WebSearch/WebFetch** (never fabricate citations). Smoke-test `<lit> --help` at setup; if the skill is
absent, tell the user and offer to install it (`cp -r <repo>/loops/literature-search ~/.claude/skills/`)
or proceed degraded. For onboarding and API keys (all optional; a free `S2_API_KEY` is recommended),
run `<lit> keys --init` — it manages the shared gitignored `keys.env` at the project root and reports
presence as booleans (secrets never enter chat). Persist the live tiers to `loop.run.yaml`.

## Initialise the sandbox
Create the layout (extra `literature/` tree only when `<literature> = on`) and write the ledger headers:
```
<sandbox_root>/
├── loop.run.yaml       ← resolved bindings (written now)
├── results.tsv         ← experiment ledger, header only (written now)
├── literature/         ← (on only)
│   ├── corpus.tsv      ← findings ledger, header only (written now)
│   ├── .cache/         ← lit_search on-disk cache
│   ├── pdfs/           ← fallback PDF reads
│   └── text/           ← extracted LaTeX section text
└── iter1/              ← created at loop start
```
`results.tsv` header (tab-separated; the `literature_basis` column exists only when `<literature> = on`):
```
iter	<metric>	status	analysis_summary	[literature_basis	]description
```

## The loop (LOOP FOREVER — until interrupted)
Iteration 1 is always the **unmodified baseline**: skip change-planning and research (no diagnostics
yet to ground a change), but still write a baseline `plan.md` and run the **mandatory analysis** — it
produces the first empirical anchor that iteration 2 builds on. Everything in `<editable_files>` is fair
game (architecture, optimizer, hyperparameters, data pipeline, loss); the only constraints are that the
code runs without crashing and finishes within `<budget>`. **Simplicity criterion**: all else equal,
simpler is better — a 0.001 gain that adds 20 lines of hacky code is not worth it; a 0.001 gain (or an
equal metric) from *deleting* code is a `keep`.

Copy this checklist each iteration and tick items off:
- [ ] **1. Look at the state.** *branches*: `git log --oneline -5`. *snapshots*: confirm `iter<N>/`
      doesn't exist. Read iter N-1's analysis summary; (on) skim `corpus.tsv` for unimplemented keepers.
- [ ] **2. Plan one change** (iteration 1: SKIP — run baseline unmodified). Grounded in iter N-1's
      analysis. See **Planning a change** below; (on) it also runs the literature step.
- [ ] **3a. Snapshot / commit, then apply the one change.** *snapshots*: create
      `iter<N>/{code_snapshot,analysis,results}/`, copy every `<editable_files>` into `code_snapshot/`,
      copy `loop.run.yaml` to `iter<N>/`, then apply. *branches*: apply, then `git commit -am "<desc>"`.
      When implementing a published/library technique, ground it in a real current example or the actual
      library in the repo (read it first) — never write the API from memory.
- [ ] **3b. Write the analysis plan BEFORE the run** + add any instrumentation it needs. See **The
      analysis plan** below.
- [ ] **4. Run the experiment**, redirecting everything (never `tee`):
      `<entrypoint> > <sandbox_root>/iter<N>/<run_log> 2>&1` (or `run_with_timeout.sh` when time-gated).
      If it overruns, kill it and treat as a crash.
- [ ] **5. Read the metric**: `grep '^<metric>:' <sandbox_root>/iter<N>/<run_log>`. If empty,
      `tail -n 50 <run_log>`, read the trace, attempt one trivial fix (typo/import); if fundamentally
      broken, log `crash` and continue.
- [ ] **6. Analyse the results** — MANDATORY, produces real artifact files. See **Analysing** below.
- [ ] **7. Log to the ledger(s)** (untracked — never commit). See **Ledger**.
- [ ] **8. Keep or revert** (the change ran this iteration). Improved per `<metric_direction>` → `keep`,
      update current-best. Equal/worse/crash → `discard`/`crash`; *branches* `git reset --hard HEAD~1`,
      *snapshots* restore `<editable_files>` from `iter<N>/code_snapshot/`. Apply the simplicity
      criterion before logging `discard`. On a crash/OOM, fix with the *minimal* change that preserves
      the intent (OOM → smaller batch + grad-accum to hold effective batch) — never mutate the
      experiment into something the plan didn't call for.
- [ ] **9. Go to step 1** — the analysis from step 6 is the primary input to the next hypothesis.

### Planning a change (step 2 — iterations 2+)
The latest analysis sets the direction; it is the master input every iteration. Decide exactly **one**
lever, grounded in iter N-1's analysis. Other vetted ideas are queued, not bundled into one run.

**State explicitly** before applying:
- the one change and which `<editable_files>` it touches;
- the **empirical anchor** — a specific file + value/pattern from iter N-1's `results/` that motivates
  it. Every non-simplification change must cite an anchor; theoretical reasoning alone is insufficient.
  A *swing* to a different architecture is anchored too (a ceiling/structural finding, e.g. "the family
  plateaus at X with headroom" or "it fails exactly on cases needing Y"), not a local pathology.
- what you predict will happen and why the finding supports it.

**When `<literature> = off`:** that anchor is the whole basis — pick the change directly from the
analysis. Before writing the plan, scan prior analyses (`ls <sandbox_root>/iter*/analysis/*.py`) so you
don't repeat a diagnostic without a comparison reason.

**When `<literature> = on`:** after fixing the anchor, *ground the change in the literature* —
- **2a. Retire drift, then consult the backlog as a cache.** If the last kept change altered the
  architecture *family* (e.g. CNN→transformer), set `result=stale` for every unimplemented keeper whose
  `scope` is a non-matching architecture tag; `scope=agnostic` keepers (schedules, weight decay,
  augmentation, init *philosophies*) survive. Then check `corpus.tsv` for an unimplemented `keep`
  targeting the analysis's direction — reuse it **only if it still passes the gate (2c) against the
  CURRENT architecture** (re-validate now; a finding that no longer applies is retired, not forced in).
- **2b. Research the direction** (the default unless 2a yielded a still-valid lever). Turn the
  analysis's limitations into questions (tie limitations to questions), record them in
  `iter<N>/questions.md`, then dispatch **research subagents** — see `roles/research-subagent.md`
  (spawn-or-degrade: real isolated subagents on Claude Code, otherwise run the research inline in this
  context) at the dial's depth/effort (see that file's dial table for `<research_scale>`).
  Level 1 = high-level (architecture fit, prior approaches, does the literature show success); research
  L1 first — if it surfaces a compelling new direction that becomes the lever. Level 2 = specific
  micro-opts (init, weight-decay dynamics, attention/cache for the sequence length, norm placement,
  schedule). *Anti-rut: a keeper passed over for ~3 iterations, or no longer on any live direction, is
  retired (`result=stale`) so it stops resurfacing.*
- **2c. Evidence gate** (re-validated; consults history). Keep/reuse a finding only if **all** hold:
  `evidence.strength` is `moderate`/`strong` (not `weak`); `applicability_to_our_setup` transfers to
  the CURRENT data/model/budget; every `files_to_edit` is within `<editable_files>` and fits `<budget>`;
  and it is **not materially equivalent to a change already logged `no-effect`/`hurt`/`crashed`** in
  `corpus.tsv` (unless the finding explains why it differs now). Tag each kept finding's **`scope`**
  (`agnostic` or the current architecture tag) so drift retirement is mechanical. Record every finding
  in `corpus.tsv` (keep or reject); kept-but-unchosen findings stay queued (`implemented=n`,
  `result=pending`).
- **2d. Choose the single lever** from the keepers (backlog + new) — highest evidence × applicability ×
  gain-per-complexity. Add to the explicit statement above: the **literature basis** (kept finding +
  citation(s)), or `none` for a pure analysis-driven simplification/ablation; and the **verify-todo**
  (the metric/log/analysis from the finding's `how_to_verify` that will tell whether it helped). The
  literature basis is *additive* grounding — it never replaces the empirical anchor.

### The analysis plan (step 3b — written BEFORE the run)
Write `<sandbox_root>/iter<N>/analysis/plan.md` as a deliverables **table**, not prose — each row is a
hard commitment: a script you will write and run, the exact output file it produces, the question it
answers, and what each outcome would look like:

| script | output file | question it answers | expected if change helped | expected if not |
|--------|-------------|---------------------|---------------------------|-----------------|

The plan **must** include at least one diagnostic dimension **not measured in any prior iteration**
(the history scan enforces this); the "expected if not" column is mandatory (it forces you to define
falsification before seeing the result). When `<literature> = on`, also include the **verify-todo**
from 2d, and periodically (and whenever a swing is on the table) a **ceiling/headroom probe** — is the
current architecture near its limit? (train/val plateau, representation collapse, structural failure).

**Instrumentation (mandatory when needed).** If a planned measurement needs a metric/log not currently
produced (e.g. per-layer grad norms, per-class accuracy, `val_loss` alongside `val_acc`) and the
producing script is in `<editable_files>`, **add that logging NOW, before the run.** Writing the plan
first is what forces this — don't discover at analysis time that the number you need wasn't logged.

### Analysing (step 6 — MANDATORY; produces real artifacts)
This is the spine of the loop. **Do not proceed to step 7 until every row in `plan.md` has a
corresponding non-empty file in `iter<N>/results/`.** Analysis that didn't write a file did not happen.
- **6a. Execute the plan.** For each row, write the script in `iter<N>/analysis/` and run it,
  redirecting to `iter<N>/results/`. Diagnostic dimensions to draw from (choose what the hypothesis and
  results call for, not a fixed rotation): gradient norms/flow, activation stats/saturation, embeddings
  (PCA/CKA/collapse), error & confusion analysis, loss dynamics & **headroom**, weight/parameter stats,
  data profiling (auditing the data itself is one of the highest-yield diagnostics), compute profiling.
- **6b. Checklist.** `ls iter<N>/results/` against `plan.md`; write and run any missing deliverable now.
- **6c. Interpret against the plan.** For each row, did it match "expected if helped" or "if not"? A
  mismatch is information, not a failure.
- **6d.** (on) **Check the verify-todo** from 2d — did the change do what the literature predicted? This
  sets the finding's `result` in step 7.
- **6e. Opportunistic follow-up.** If something unexpected appears (odd loss shape, a consistently-wrong
  class, a layer behaving differently), write one more script to chase it — the most novel findings
  come from here.

Write a concise **analysis summary** (3–8 bullets): what you examined, the single most important
finding (expected or not), (on) whether the architecture is near its ceiling, and the specific
**empirical anchor** (file + value/pattern) that will motivate the next plan.

**Empirical-justification gate.** Before step 7, you must be able to complete this sentence with a real
file reference and a concrete value/pattern — not a theoretical argument:
> *"The next change will be X because the analysis showed Y (from `results/<file>`, value/pattern Z)."*

If you cannot, go back to 6a and run more analysis until you can. Proceeding without an empirical anchor
is the exact failure this loop is built to prevent.

**Forward-looking instrumentation.** After analysing, ask *"what would I wish I had logged?"* — if the
producing script is in `<editable_files>`, add it now (best-epoch checkpoint, extra per-epoch
diagnostics) so future analyses are richer. Adding instrumentation is a valid iteration on its own.

## Ledger
`<sandbox_root>/results.tsv`, tab-separated, never commas in free text. The `literature_basis` column
is present only when `<literature> = on`. Status ∈ {`keep`, `discard`, `crash`}; use `0.000000` for the
metric on crashes. For `branches`, the first column is the 7-char commit instead of the iter number.
```
iter	<metric>	status	analysis_summary	literature_basis	description
1	0.6320	keep	grad norms clean; no pathologies	none (baseline)	baseline
2	0.7050	keep	overfit on small data per val/train gap	Dosovitskiy 2020 ViT — aug+warmup	add RandAug + 3-ep warmup
3	0.7010	discard	warmup ok but LR too high; unstable	Loshchilov 2017 — decoupled wd	switch to AdamW wd=0.05
```
**`<sandbox_root>/literature/corpus.tsv`** (only when `<literature> = on`; tab-separated): a backlog of
vetted findings reused across iterations instead of re-researched. The orchestrator (not the subagent)
writes it and applies the final evidence gate, so verdicts are consistent across questions.
```
iter	level	paper_id	title	scope	relevance	verdict	implemented	result
2	649def…	1	An Image is Worth 16x16 Words	vit	high	keep	y	helped
3	a1c2…	2	Decoupled Weight Decay Regularization	agnostic	high	keep	y	no-effect
4	7d9e…	1	ConvNeXt blocks	cnn	low	keep	n	stale
```
Columns: `level` ∈ {1,2}; `scope` = `agnostic` or an architecture tag (drives drift retirement);
`relevance` ∈ {high,med,low}; `verdict` ∈ {keep,reject}; `implemented` ∈ {y,n}; `result` ∈
{helped,no-effect,hurt,pending,stale} (`stale` = retired on drift/expiry). Last row: a CNN keeper
retired after the architecture moved to ViT; the `agnostic` weight-decay finding survives drift.

Report the **best** iteration (not necessarily the last) when summarising progress. Do not commit
`results.tsv` or `literature/` to git — leave them untracked.

## Constraints
- **Only edit files in `<editable_files>`** — confirm before every edit, because everything else is
  read-only ground truth and the eval harness defines `<metric>`.
- **One change per iteration**, so each metric delta is attributable to one lever.
- **Analysis is mandatory every iteration** and must produce real files in `results/` (each `plan.md`
  row → a non-empty output file). A change with no analysis behind it is not allowed — it degrades the
  loop into blind iteration (or, when on, recipe-following that plateaus blind).
- **Every non-simplification change must cite an empirical anchor** (file + value/pattern) from that
  analysis; when on, the literature basis is additive, never a substitute. Record both in `results.tsv`.
- Do not install packages or add dependencies the project lacks; helper scripts (and the lit helper)
  stay stdlib-only.
- Always redirect training output to `<run_log>`; never use `tee`. The sandbox is self-contained — no
  `../` escapes.
- (on) Never print or commit API keys — `keys` reports presence only; `keys.env` lives at the project
  root and must stay gitignored. When a lit tool returns `{"error","fallback"}`, degrade to
  WebSearch/WebFetch; never fabricate citations — every cited paper must come from a real tool result.

## Stops
This loop runs **forever until the human interrupts it** — do not pause to ask "should I continue?" or
"is this a good stopping point?". The human may be away and expects autonomous work indefinitely. A
working result is the start of the next iteration, not the end. If you run out of ideas: re-read the
in-scope files for missed angles, go deeper on the analysis (gradients/activations/embeddings/errors
always surface something), combine previous near-misses from `results.tsv`, try more radical changes —
and when on, **go back to the literature**: crawl citation graphs deeper, read methodology of recent
work citing your approach, mine `corpus.tsv` for queued keepers, and combine recipes across papers.

© gaasher · MIT · gaasher/Agent-Loop-Skills

Similar skills

All skills for AI and ML →

From the same set26

The whole set →
swe-loop swe-loop is a Claude Code skill that breaks coding requests into tasks, has an Engineer implement them, and QA verify tests and quality. CX Codex +1 ★ 185repo stars scientific-writer scientific-writer refines scientific drafts through iterative critique and peer review. Multiple specialist judges evaluate prose, data, and figures until quality clears the bar. CX Codex +1 ★ 185repo stars alpha-evolve The alpha-evolve skill explores many model-code variants in parallel, scores them through training runs, and keeps diverse strong candidates. Use it for evolutionary search instead of sequential edits. CX Codex +1 ★ 185repo stars blue-team blue-team runs a fix loop over a concrete list of failures: one root-cause class per iteration, a revert on any regression, and a pull request at the end. CX Codex +1 ★ 185repo stars scientific-figure Builds publication-ready scientific figures from data or prompts through an adversarial generator-critic loop evaluated against a strict rubric. CX Codex +1 ★ 185repo stars tabular-cleanup The tabular-cleanup skill cleans messy CSV, TSV, Parquet, Excel, or JSON tables against an agreed data contract. Use it for iterative cleanup with explicit checks and a change log. CX Codex +1 ★ 185repo stars exploratory-autoresearch exploratory-autoresearch runs an autonomous ML experiment loop that samples wildly different designs first, then switches to picking one move per round. CX Codex +1 ★ 185repo stars agent-loops Run agent loops that propose a change, score the result, and keep or revert it. Useful when your task has a measurable feedback signal. ★ 185repo stars

Project sponsors

Companies that keep this analytics open