experiment-code — Claude Code skill: ML experiment code

What experiment-code does

The skill has four actions. generate plans every run first — hyperparameter sweeps, ablations, baselines — then writes self-contained code inside the project directory, with no imports from reference repos. improve reads the current code and results, makes targeted edits, and keeps the best-scoring variant. debug reads the error message and applies a minimal fix, up to 4 retries before switching approach. plot pulls numbers from run_*/final_info.json and builds publication figures.

The project layout is fixed: experiment.py, plot.py, notes.txt, and run_1, run_2 folders holding final_info.json. Each run goes through python experiment.py --out_dir=run_i. Placeholder code such as pass or raise NotImplementedError is off the table, and the data has to be real rather than toy. PyTorch or scikit-learn are the preferred frameworks.

A few rules catch people out. The method must not land at 0% accuracy, so accuracy calculations get checked — a zero usually means a bug, not a failed idea. Seeds are required for reproducibility. Before each experiment the code needs a print statement saying what the results are supposed to show. One more thing for long logs: the error message is truncated to the last 1500 characters.