Skip to main content
skill CX Codex

autoresearch · akillness — Claude Code skill: what it does, how to install

autoresearch — a skill for Claude Code runs Karpathy-style closed-loop ML exploration. Enforces a strict testing contract where the agent modifies only training code and discards regressions.

MIT 349 installs ★ 61 updated 2026-10-24

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add akillness/jeo-skills --skill autoresearch -a claude-code
Open on GitHub

What autoresearch does

The skill routes the assistant into one of five focused modes: initial environment setup, authoring the program.md charter, running a bounded training search, analyzing results.tsv, or handling constrained VRAM budgets.

It enforces an immutable evaluation contract. The data script prepare.py and the 300-second execution window remain strictly untouched. The agent edits solely train.py, checks val_bpb, retains commits that strictly improve the metric, and executes git reset HEAD~1 on ties or regressions.

Execution requires the uv package manager and a dedicated machine with CUDA or MLX GPU support. It deliberately avoids prompt optimization or app-level tracing, routing such requests toward tools like LangSmith or Promptfoo.

What it helps with

  • Configure a fresh GPU environment and record the baseline val_bpb
  • Draft search directions and constraints in program.md
  • Execute unattended 300-second training iterations with automatic git rollbacks
  • Filter and rank kept runs from results.tsv

Example prompts for the agent

  • Set up the autoresearch repo on this machine and verify hardware readiness
  • Write a program.md file outlining prioritized changes for train.py
  • Review results.tsv and display the top runs marked as keep

How to install autoresearch

1Skills CLI (Vercel)

npx skills add akillness/jeo-skills --skill autoresearch -a claude-code

2Manually: put the files in place

~/.claude/skills/autoresearch/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/autoresearch/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about autoresearch

How to install autoresearch in Claude Code?

Run npx skills add akillness/jeo-skills --skill autoresearch -a claude-code. This installs the definition to ~/.claude/skills/autoresearch/SKILL.md.

What does autoresearch do?

It manages an autonomous ML research loop that iterates on train.py, maintains a fixed 300-second run budget, and keeps changes only when val_bpb drops.

What is required to run autoresearch?

You need a GPU-enabled system with CUDA or MLX support, the uv tool installed, and a local clone of karpathy/autoresearch.

Source text

SKILL.md · MIT
---
name: autoresearch
description: >-
  Run Karpathy-style autonomous ML search on a real training repo: choose the right mode (setup,
  program.md, bounded loop, results interpretation, or constrained-hardware adaptation), preserve
  the immutable prepare.py / 300-second / val_bpb contract, and route prompt/skill eval work away
  to LangSmith, Promptfoo, Braintrust, or skill-autoresearch.
allowed-tools: Bash Read Write Edit Glob Grep WebFetch
metadata:
  tags: autoresearch, ml-experiments, autonomous-research, karpathy, gpu, train, val-bpb, overnight, ratcheting
  version: 1.2.0
  source: "https://github.com/karpathy/autoresearch"
  license: MIT
---












# autoresearch

Autoresearch is a **closed-loop ML experimentation workflow**:
- human writes `program.md`
- agent edits `train.py`
- `prepare.py` stays fixed
- every run gets the same 300-second budget
- lower `val_bpb` wins
- regressions get reverted

This skill should behave like a **routing-first front door**, not a giant tutorial. Pick the user's mode, enforce the immutable-harness rules, then hand them to the smallest useful script or reference.

## When to use this skill

- Set up `karpathy/autoresearch` on a real GPU machine
- Write or refine `program.md` before a session
- Run a bounded overnight `train.py` search loop
- Interpret `results.tsv` after a session
- Adapt the workflow to tighter VRAM constraints without invalidating comparisons
- Explain the ML-specific boundary between `autoresearch` and nearby eval tooling

## Do not use this skill when

- The user wants to optimize a `SKILL.md`, prompt, or repo-local workflow with frozen prompts/evals — use `skill-autoresearch`
- The user wants app-level tracing, dataset-backed LLM evals, feedback review, or observability — use LangSmith, Braintrust, Weave, Promptfoo, or similar tools
- The job does not involve a real training repo, `program.md`, `train.py`, fixed runtime budget, and `val_bpb` keep/revert ratcheting
- The user is really asking for a paper survey, general benchmark scan, or literature review with no intention to run the training loop

## Core boundary

| Concern | `autoresearch` owns | Route elsewhere |
|---------|---------------------|-----------------|
| Mutable target | `train.py` in a real training repo | prompts, app configs, `SKILL.md`, product behavior |
| Fixed evaluator | `prepare.py`, validation shard, `TIME_BUDGET=300`, chosen `MAX_SEQ_LEN` / `EVAL_TOKENS` for the session | prompt/eval datasets, app scorecards, observability dashboards |
| Acceptance rule | keep only lower `val_bpb`; revert ties/regressions | human review queues, app-level release gates |
| Main artifacts | `program.md`, `results.tsv`, kept/discarded commits | prompt suites, traces, feedback datasets |

If that boundary does not fit, do not stretch this skill.

## Required intake packet

Before acting, identify:
1. **Mode** — setup, `program.md`, run loop, results interpretation, or constrained hardware
2. **Repository state** — cloned or not, dependencies installed or not
3. **Hardware state** — GPU / VRAM / CUDA / MLX / Windows path
4. **Session state** — first baseline, active loop, or completed run
5. **Constraint state** — target VRAM ceiling, whether `prepare.py` has already been frozen for this session

## Instructions

### Step 1: Pick exactly one operating mode

Choose the smallest mode that answers the request:

1. **Setup readiness**
   - install `uv`
   - clone repo
   - sync dependencies
   - verify GPU/CUDA/uv with `scripts/check-hardware.sh`
   - run the first baseline experiment

2. **`program.md` authoring**
   - write or refine the human research charter
   - record current baseline `val_bpb`
   - prioritize hypotheses
   - list what has already been tried
   - freeze constraints before the loop starts

3. **Bounded run loop**
   - confirm the evaluator is already fixed
   - use `train.py` as the only mutable search surface
   - run the loop with keep/revert discipline
   - log every experiment to `results.tsv`

4. **Results interpretation**
   - summarize best kept runs
   - identify repeated failures or crash patterns
   - extract what belongs in the next `program.md`
   - distinguish genuine gains from one-off anomalies

5. **Constrained-hardware adaptation**
   - set `MAX_SEQ_LEN` and `EVAL_TOKENS` before the session
   - keep them unchanged once the session starts
   - adjust model/search strategy instead of cheating the evaluator mid-run
   - route to community forks when CUDA assumptions do not hold

Do **not** answer all five modes at once unless the user explicitly asked for a full end-to-end walkthrough.

### Step 2: Re-state the immutable harness

Every mode must preserve these rules:

- `program.md` is human-authored and read-only during a session
- `train.py` is the main mutable search surface
- `prepare.py` is read-only once the session starts
- `TIME_BUDGET=300` stays fixed
- `val_bpb` is the main keep/revert metric
- `results.tsv` is append-only
- dependency set in `pyproject.toml` stays locked

If the user wants to change the evaluator, start a **new comparison track**, not the current session.

### Step 3: Execute the chosen mode

#### Mode A — Setup readiness

Use this path when the repo is not yet runnable.

```bash
# Prerequisite: uv installed (e.g. `brew install uv` or `python3 -m pip install --user uv`)
git clone https://github.com/karpathy/autoresearch
cd autoresearch
uv sync
bash scripts/check-hardware.sh
uv run prepare.py
uv run train.py > run.log 2>&1
grep "^val_bpb:\|^peak_vram_mb:" run.log
```

Success condition: one baseline run completes and prints both `val_bpb` and `peak_vram_mb`.

#### Mode B — `program.md` authoring

Use this path when the loop exists but direction is weak.

Minimum sections:
- goal tied to lower `val_bpb`
- current baseline `val_bpb`
- directions to explore in priority order
- what has been tried already
- constraints: `TIME_BUDGET=300`, no `prepare.py` mutation, no new packages, VRAM ceiling, one meaningful change per experiment

For fuller templates and update patterns, use `references/program-md-guide.md`.

#### Mode C — Bounded run loop

Use this path only after setup and `program.md` are ready.

Loop contract:
1. read `program.md` + current `train.py`
2. form one hypothesis
3. edit `train.py`
4. commit
5. run one 300-second experiment
6. extract `val_bpb`
7. keep if improved, otherwise `git reset HEAD~1`
8. append result to `results.tsv`

Typical commands:

```bash
bash scripts/run-experiment.sh
bash scripts/run-loop.sh --max 20 --desc "session-1"
```

Do not encourage multi-change hero rewrites. Clean ablations matter more than flashy edits.

#### Mode D — Results interpretation

Use this path after a completed run or checkpoint.

Helpful commands:

```bash
bash scripts/show-results.sh --top 10
awk -F'\t' '$4=="keep"' results.tsv | sort -t$'\t' -k2 -n
awk -F'\t' '{print $4}' results.tsv | sort | uniq -c
```

Summarize only four things: best gains, repeated failures, what should move into `What Has Been Tried`, and the next narrow experiment family.

#### Mode E — Constrained-hardware adaptation

Use this path when VRAM, platform, or runtime constraints dominate.

Rules:
- choose `MAX_SEQ_LEN` and `EVAL_TOKENS` **before** the session
- never change them mid-session
- lower model/search ambition before mutating the evaluator
- prefer route-outs to community forks for Apple Silicon / non-CUDA paths

For concrete values and troubleshooting, use `references/hardware-config.md`.

### Step 4: Route out aggressively when the request is adjacent

Route out when:
- the user wants to optimize instructions, prompts, or repo-local skills → `skill-autoresearch`
- the user wants app-level traces, feedback review, observability, or online/offline eval dashboards → LangSmith / Braintrust / Weave / Promptfoo
- the user wants general literature synthesis rather than a runnable ML loop → research or survey tooling

### Step 5: Keep the heavy detail in support files

Use support files instead of re-explaining everything inline:
- `references/operating-modes-and-route-outs.md` — fast routing table, minimal response shape, and handoff logic
- `references/architecture.md` — immutability contract, file map, metric rationale
- `references/program-md-guide.md` — templates and update rules
- `references/hardware-config.md` — VRAM tables and platform troubleshooting
- `scripts/*.sh` — runnable setup / loop / reporting helpers

## Available scripts

Run from inside the autoresearch repository directory:

| Script | Purpose | Usage |
|--------|---------|-------|
| `setup.sh` | One-time environment setup | `bash scripts/setup.sh [--seq-len 512]` |
| `run-experiment.sh` | Single 5-minute experiment + metric extraction | `bash scripts/run-experiment.sh` |
| `run-loop.sh` | Autonomous loop: run → keep/revert → repeat | `bash scripts/run-loop.sh [--max 20]` |
| `show-results.sh` | Human-readable `results.tsv` report | `bash scripts/show-results.sh [--top 10]` |
| `check-hardware.sh` | GPU/CUDA/uv readiness check (JSON output) | `bash scripts/check-hardware.sh` |

## References

Detailed documentation in `references/`:

| File | Contents |
|------|----------|
| `references/operating-modes-and-route-outs.md` | Mode picker, adjacency boundaries, and minimal output contract |
| `references/architecture.md` | System design, immutability contract, git ratcheting, metric rationale |
| `references/program-md-guide.md` | How to write and update effective `program.md` directives |
| `references/hardware-config.md` | VRAM settings by GPU, memory optimization, platform troubleshooting |

## Examples

### Example 1: First 40GB GPU session

Request: “Help me run Karpathy autoresearch on a 40GB GPU.”

Expected behavior:
- choose **Setup readiness** first
- verify hardware and dependencies
- run one baseline experiment
- route to `program.md` authoring only after the baseline exists

### Example 2: User wants to optimize a skill instead

Request: “Can autoresearch help me improve this `SKILL.md` with binary evals?”

Expected behavior:
- route out immediately to `skill-autoresearch`
- explain that this skill is for real ML training search on `train.py`

## Best practices

1. **Start with the smallest mode that fits** — setup, authoring, run loop, interpretation, or hardware adaptation
2. **Baseline before bravado** — confirm one successful run before talking about overnight loops
3. **Freeze the evaluator before the session** — `prepare.py`, `TIME_BUDGET`, `MAX_SEQ_LEN`, and `EVAL_TOKENS` must stay comparable
4. **One meaningful experiment at a time** — ablations beat mystery bundles
5. **Keep `results.tsv` append-only** — discarded runs are still evidence
6. **Push deep detail into references/scripts** — the front door should classify and route, not duplicate every table
7. **Route adjacent jobs away early** — prompt/app eval and `SKILL.md` optimization are different lanes

## References

- [GitHub — karpathy/autoresearch](https://github.com/karpathy/autoresearch)
- [Karpathy — A Recipe for Training Neural Networks](https://karpathy.github.io/2019/04/25/recipe/)
- [MLflow Tracking](https://mlflow.org/docs/latest/ml/tracking/)
- [Weights & Biases Tracking](https://docs.wandb.ai/guides/track/)
- [MIT License](https://github.com/karpathy/autoresearch/blob/master/LICENSE)

© akillness · MIT · akillness/jeo-skills

Similar skills

All skills for AI and ML →

autoresearch by other authors comes first.

autoresearch official luiscantero autoresearch — a GitHub Copilot skill that runs an autonomous experiment loop: it edits code, tests, measures a metric, and keeps only what improves it. The project must be a git repository with terminal access. CX Codex +1 3,400installs autoresearch uditgoenka The autoresearch skill changes code, checks it against a metric, then keeps or discards the result. Separate commands cover debugging, security audits, and other workflows. CX Codex +1 2,700installs autoresearch Yeachan-Heo autoresearch runs one mission through repeated change-and-evaluate cycles, writing each result to JSON and a markdown decision log. Useful when the work needs a long recorded search rather than one attempt. CX Codex +1 654installs autoresearch Orchestra Research autoresearch manages autonomous research projects using a two-loop cycle for rapid experiment execution and periodic synthesis. CX Codex +1 1,200installs autoresearch AgriciDaniel autoresearch runs a bounded research loop with web access, drafts a cited report, and lets you review before filing results. Use when you need deep investigation with fact-checking built in. CX Codex +1 572installs tournament-autoresearch gaasher The tournament-autoresearch skill pits architecture proposals against each other before an ML experiment runs. It suits projects where you want an agent to choose one idea and learn from its measured result. CX Codex +1 ★ 185repo stars exploratory-autoresearch gaasher exploratory-autoresearch runs an autonomous ML experiment loop that samples wildly different designs first, then switches to picking one move per round. CX Codex +1 ★ 185repo stars serverless-modal wanshuiyin serverless-modal runs GPU training and inference on Modal without SSH or Docker configuration. Prepares launcher scripts and estimates compute cost right inside your agent. CX Codex +1 515installs autoresearch-agent Alireza Rezvani autoresearch-agent runs repeated file experiments against a measurable metric, keeping gains and reverting failures. Use it for code, tests, prompts, or content. CX Codex +1 589installs wandb-autoresearch wandb wandb-autoresearch runs bounded machine learning trials via W&B Launch, smoke-tests real jobs, compares metrics, and tracks research state across sessions. CX Codex +1 387installs mlx sanjeed5 Guides Cursor to write high-performance Apple Silicon code with MLX, enforcing modular structure, unified memory usage, and explicit lazy evaluation. CX Codex +1 ★ 3,589repo stars

From the same set99

The whole set →
skill-standardization skill-standardization is a Claude Code skill that validates SKILL.md files against the Agent Skills spec and standardizes trigger-focused frontmatter. It also removes catalog drift across repositories. CX Codex +1 353installs system-environment-setup system-environment-setup is a Claude Code skill for creating reproducible project environments across machines: it pins runtime versions, sets up local services, and generates bootstrap scripts and devcontainer configs. CX Codex +1 352installs npm-git-install npm-git-install picks a safe delivery path for a Node package coming from a branch, commit, fork or private repo. Useful when the question is whether to pull from Git at all. CX Codex +1 351installs game-performance-profiler game-performance-profiler turns Unity and Unreal frame-time complaints into one brief: which capture to trust, where the bottleneck sits, what to measure next. CX Codex +1 350installs genkit Guides Firebase AI features by picking between the client AI Logic SDK and server-side Genkit flows. CX Codex +1 348installs vercel-deploy This skill operates Vercel deployments for linked projects, from preview URLs and production releases to aliases, domains, environment sync, and rollback checks. CX Codex +1 348installs react-grab react-grab is a Claude Code skill for sending a React element’s component name, file path, and HTML from a live browser to an AI agent. It activates for browser-based component lookup and clipboard or MCP setup. CX Codex +1 346installs playwriter playwriter — a skill for Claude Code, Codex, Cursor, and other agents that connects browser automation to an already-running Chrome session — with active cookies, logins, and extensions intact. CX Codex +1 345installs

Project sponsors

Companies that keep this analytics open