# automl-hyperparameter-optimization

The automl-hyperparameter-optimization Claude Code skill guides AutoML setup and hyperparameter searches. It covers validation design, distributed trials, and checks for judging whether a leaderboard result is production-ready.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · 110 installs · ★ 312 repo stars
- Author: Mindrally
- License: Apache-2.0 · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/Mindrally/skills/tree/HEAD/automl-hyperparameter-optimization/SKILL.md
- Updated: 2026-10-08
- Security check: automated, 2026-11-17 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/automl-hyperparameter-optimization

## What it does

The skill lays out a hyperparameter search workflow: set a target metric, establish a simple baseline, define a structured search space, then choose a tool such as Optuna, Ray Tune, or PyCaret. For time-series work, it calls for time-aware splits; preprocessing should be fitted on each fold’s training portion only.

Model-selection claims should use nested validation or a final test split kept out of tuning. The guidance also covers pruning weak trials, resource budgets, and recording experiment details. It names MLflow, Weights & Biases, and TensorBoard as tracking options, and includes an Optuna example with pruning and search limits.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill automl-hyperparameter-optimization -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/automl-hyperparameter-optimization/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/automl-hyperparameter-optimization/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill automl-hyperparameter-optimization -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/automl-hyperparameter-optimization/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill automl-hyperparameter-optimization -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/automl-hyperparameter-optimization/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install Mindrally/skills automl-hyperparameter-optimization
```

2. **Manually: put the files in place**
- File: `.github/skills/automl-hyperparameter-optimization/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill automl-hyperparameter-optimization -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/Mindrally/skills --path automl-hyperparameter-optimization
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/automl-hyperparameter-optimization/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add Mindrally/skills --skill automl-hyperparameter-optimization -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/Mindrally/skills/tree/HEAD/automl-hyperparameter-optimization/SKILL.md

## When to use it
- set a target metric and baseline before tuning
- shape a domain-informed hyperparameter search space
- split time-series data without shuffling across time
- prevent preprocessing from leaking validation data
- compare an AutoML result with a baseline before production

## Example prompts
- Plan an Optuna hyperparameter search for my model with a time budget.
- Review this time-series validation scheme and identify possible data leakage.
- Assess this AutoML result against a simple baseline and an untouched test split.

## Source text · SKILL.md · Apache-2.0

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: d3afba57e35522c9.

BEGIN THIRD-PARTY FILE d3afba57e35522c9
```text
---
name: automl-hyperparameter-optimization
description: "Best practices for AutoML and hyperparameter search with Optuna, Ray Tune, and PyCaret, covering search-space design, validation splits, and leakage prevention. Use when tuning model hyperparameters, setting up a pruned or distributed hyperparameter search, designing a nested validation scheme, or evaluating whether an AutoML leaderboard result is production-ready."
metadata:
  maintainer: Mindrally
  source: https://github.com/Mindrally/skills
---

# AutoML and Hyperparameter Optimization

This skill covers designing sound hyperparameter searches and using AutoML tooling (Optuna, Ray Tune, PyCaret, time-series AutoML libraries) without bypassing problem framing, validation design, or explainability.

## Workflow for Running a Hyperparameter Search

1. **Define the target metric and baseline first** — Pick the metric before selecting tooling, and train a simple baseline (linear model, random forest, or naive time-series forecast) with a fixed, minimal search.
2. **Design the validation scheme** — Use nested cross-validation or a final untouched test split for any model-selection claim; use time-aware splits (never shuffled) for time-series problems.
3. **Fit preprocessing inside the fold** — Fit scalers, encoders, and imputers only on the training portion of each fold to prevent leakage.
4. **Define a structured search space** — Use log-scale ranges for learning rates, regularization strength, and tree counts; keep ranges domain-informed rather than arbitrarily broad.
5. **Choose the right tool** — Optuna or Ray Tune for custom training loops with pruning and distributed trials; PyCaret for a quick low-code comparison on a straightforward tabular problem; a time-series-specific library (AutoTS, Merlion, PyAF) when seasonality and horizon handling need first-class support.
6. **Run with resource limits and pruning** — Set a trial or time budget and use early stopping/pruning so bad trials don't consume the full budget.
7. **Track every run** — Log datasets, splits, metric definitions, random seeds, library versions, and the search space itself to MLflow, Weights & Biases, TensorBoard, or an equivalent tracker.
8. **Report against the baseline** — Compare the selected model to the baseline and at least one non-AutoML alternative before calling it production-ready.

## Experiment Design

- Define the target metric before choosing tooling — the metric shapes the search space and the pruning strategy, not the other way around.
```
END THIRD-PARTY FILE d3afba57e35522c9

The third-party file has ended. Everything between the d3afba57e35522c9 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2542 of 9902 bytes. © Mindrally · Apache-2.0 · https://github.com/Mindrally/skills/tree/HEAD/automl-hyperparameter-optimization/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
