prompt-optimizer · haabe — Claude Code skill: A/B test CLAUDE.md edits
Runs your CLAUDE.md edits against two eval splits and shows whether the gain is real or just fitting the test set.
npx skills add haabe/mycelium --skill prompt-optimizer -a claude-codeWhat prompt-optimizer does
The skill moves in steps. baseline fires two eval splits through /mycelium:eval-runner and writes a snapshot to .claude/optimization/baseline.json — timestamp, CLAUDE.md hash, per-category metrics. You then describe a hypothesis in .claude/optimization/variants/<variant>.md and call test <variant>.
Two splits carry the point. The optimization split is the hill-climbing signal, holdout checks whether the win travels to other tasks. When optimization goes up and holdout goes down, the skill flags overfitting. Nothing reverts on its own; you decide. report lays every variant out in a table with both splits side by side and an Overfit? column.
The skill is built to run inside the Mycelium framework. On its own it skips the canvas state, theory gates and harness behavior it assumes. Install the framework first: /plugin install mycelium@haabe-mycelium.
What it helps with
- check whether a CLAUDE.md edit actually moves eval results
- catch a variant that only fits one test split
- save a snapshot of current metrics before experimenting
- compare several instruction variants in one table
- keep a clean winning run as a reference exemplar
Example prompts for the agent
- Capture a baseline for the current CLAUDE.md
- Test the
strict-toolsvariant and compare it to baseline - Build the report across all variants and flag any overfitting
How to install prompt-optimizer
1Skills CLI (Vercel)
npx skills add haabe/mycelium --skill prompt-optimizer -a claude-code2Manually: put the files in place
~/.claude/skills/prompt-optimizer/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/prompt-optimizer/
1Skills CLI (Vercel)
npx skills add haabe/mycelium --skill prompt-optimizer -a codex2Manually: put the files in place
.agents/skills/prompt-optimizer/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add haabe/mycelium --skill prompt-optimizer -a cursor2Manually: put the files in place
.cursor/skills/prompt-optimizer/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install haabe/mycelium prompt-optimizer2Manually: put the files in place
.github/skills/prompt-optimizer/SKILL.md1Skills CLI (Vercel)
npx skills add haabe/mycelium --skill prompt-optimizer -a gemini-cli2Platform CLI
gemini skills install https://github.com/haabe/mycelium --path plugins/mycelium/skills/prompt-optimizerManually: put the files in place
.devin/skills/prompt-optimizer/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add haabe/mycelium --skill prompt-optimizer -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about prompt-optimizer
How do I install prompt-optimizer in Claude Code?
Run npx skills add haabe/mycelium --skill prompt-optimizer -a claude-code. The skill lands at ~/.claude/skills/prompt-optimizer/SKILL.md.
What does prompt-optimizer do?
It runs instruction edits through two eval splits, compares them against a stored baseline, and marks variants where the optimization split improves while holdout drops.
Does prompt-optimizer work without Mycelium?
No. It is designed to run inside the Mycelium framework and relies on its canvas state, theory gates and harness. Install the framework with /plugin install mycelium@haabe-mycelium.
Source text
SKILL.md · MIT
--- name: prompt-optimizer description: "A/B test CLAUDE.md instruction changes against eval benchmarks. Capture baselines, test variants, compare results." metadata: instruction_budget: "17" framework_dependency: "mycelium" framework_dependency_note: "This skill is designed to run within the Mycelium framework (https://github.com/haabe/mycelium). Standalone use will skip the canvas state, theory gates, and harness behavior the skill assumes. Install: /plugin install mycelium@haabe-mycelium." --- # Prompt Optimizer Systematically improve Mycelium instructions through measurement. Adapted from n-trax. ## Commands ### `baseline` -- Capture current performance 1. Run `/mycelium:eval-runner run-split optimization` — record as optimization scores 2. Run `/mycelium:eval-runner run-split holdout` — record as holdout scores 3. Record both to `.claude/optimization/baseline.json`: timestamp, CLAUDE.md hash, optimization metrics, holdout metrics, overall and per-category metrics ### `test <variant>` -- Test a variant 1. Read variant from `.claude/optimization/variants/<variant>.md` 2. Apply the CLAUDE.md changes described 3. Run `/mycelium:eval-runner run-split optimization` — this is the hill-climbing signal 4. Run `/mycelium:eval-runner run-split holdout` — this validates generalization 5. Store results in `.claude/optimization/results/<variant>.json` 6. Compare against baseline. Flag **overfitting** if optimization improves but holdout degrades. 7. Do NOT auto-revert -- let user decide ### `report` -- Compare all variants Generate comparison table with split-aware columns: ``` | Variant | Opt Pass Rate | Holdout Pass Rate | Delta Opt | Delta Holdout | Overfit? | Decision | ``` Flag `Overfit? = YES` when optimization delta is positive but holdout delta is negative. ### `exemplar <eval-name>` -- Capture winning trajectory After a clean eval win (1 iteration, fast), save the approach to `.claude/optimization/exemplars/`. ## Workflow 1. Capture baseline 2. Hypothesize an instruction improvement 3. Document in variants/ directory 4. Test the variant 5. Compare via report 6. Keep or revert based on data 7. Capture exemplars from clean wins
© haabe · MIT · haabe/mycelium
Similar skills
All skills for Agent setup: memory, context, orchestration →prompt-optimizer by other authors comes first.
