bench-experiment — Claude Code command for benchmark runs
bench-experiment is a Claude Code command that runs a Conduit benchmark round and compares results with earlier sweeps to measure changes across selected cells.
MIT ★ 110 updated 2026-10-19
.claude/commands/bench-experiment.mdWhat bench-experiment does
The command prepares benchmark cells, dispatches drivers, captures metrics, and compares work, oracle, and wall-clock results with earlier runs. It then records the round in speckit-sandboxes, including stats.jsonl, history.jsonl, REPORT.md, run data, and reviews.
Before baking cells, it checks for the indexing marker and active Spotlight work. If the marker is missing or Spotlight is busy, the round waits while the user runs the specified mdutil command. Judges must wait until capture finishes, then run in small batches. Running them alongside capture has previously brought the machine down.
Install it by saving the command file at .claude/commands/bench-experiment.md. The experiment description can include options for sizes, modes, extension version, spec-kit version, driver model, and GUI preamble. When comparing the stock arm with Companion, use work: overall includes lifecycle capture, which the stock arm does not have.
What it helps with
- compare benchmark results with earlier sweeps
- test an unreleased extension build across selected cell sizes
- capture work, oracle, and wall-clock metrics
- save a benchmark round and its report in
speckit-sandboxes
Example prompts for the agent
- Run a benchmark experiment called “hook anchors” across all sizes and compare it with earlier sweeps.
- Repeat the baseline using
--ext codeand report the metrics. - Compare the Companion and stock arms using the
workmetric.
How to install bench-experiment
Manually: put the files in place
.claude/commands/bench-experiment.mdCall it with /bench-experiment. For all projects use ~/.claude/commands/
Manually: put the files in place
~/.codex/prompts/bench-experiment.mdCall it with /prompts:bench-experiment. Codex marks this format deprecated in favour of skills
Manually: put the files in place
.github/prompts/bench-experiment.prompt.mdManually: put the files in place
.gemini/commands/bench-experiment.tomlGemini stores commands as TOML: the prompt text goes into the prompt field
Questions about bench-experiment
How do I install bench-experiment in Claude Code?
Save the command file to .claude/commands/bench-experiment.md.
What does bench-experiment do?
It runs a Conduit benchmark round, compares its metrics with earlier sweeps, and records the results and report in speckit-sandboxes.
Why use work to compare the stock arm with Companion?
overall counts lifecycle capture, which the stock arm does not have. work compares correctness and rubric across the arms.
Source text
bench-experiment.md · MIT
--- allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash(node "$SANDBOXES_REPO"/bench/sync-templates.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/run-all.mjs:*), Bash(npm run compile:*), Bash(git -C:*), Bash(date:*), Bash(ps:*), Agent, Workflow, AskUserQuestion description: Run one full benchmark round on Conduit and report it against previous sweeps --- ## Your task Run one complete round of the benchmark and report what changed. The user says "run an experiment and bench it" and expects the whole thing without being asked for details. The harness lives in the `bench/` folder of the sibling [`speckit-sandboxes`](https://github.com/alfredoperez/speckit-sandboxes) repo and the app it measures in [`conduit`](https://github.com/alfredoperez/conduit). `bench/README.md` there is the reference; this command is the procedure. ### Paths Shell state does not persist between Bash calls, so start each one with this line: ```bash . .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)" ``` It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable. ### 1. Name the sweep `$ARGUMENTS` is what the experiment is testing, plus optional flags: | Flag | Default | What it does | |---|---|---| | `--sizes easy,hard` | all four | which sizes the round covers | | `--modes speckit,companion` | all arms | which arms to run | | `--ext code\|latest\|<tag>` | `code` | `code` = the working tree, `latest` = the published build, or a release tag to backtrack to | | `--speckit latest\|keep` | `latest` | pin or refresh spec-kit itself | | `--model sonnet\|opus` | the session's | model the driver agents run on | | `--preamble on\|off` | `off` | whether drivers are handed the GUI preamble | **`--preamble on` is a promise the drivers have to keep.** The bake records it, nothing enforces it: a round baked `on` whose drivers were handed nothing has a column that lies, which is worse than no column. When it is on, every driver prompt starts with the output of `node "$SANDBOXES_REPO"/bench/preamble.mjs creation <speckit|companion> <specDir>` — the same text the GUI sends — and the arm decides which workflow name it takes. When it is off, drivers dispatch the command bodies alone. Pass `--sizes`/`--modes` straight through to `sync-templates.mjs`, `prep` and `capture` — the same list at every step, or the round measures cells it never ran. Pass `--model` as the `model` option on each driver Agent. Everything else in `$ARGUMENTS` is what the experiment is testing. Turn it into a short label — "hook anchors", "living specs off", "fresh baseline". If there is nothing to name, use today's date and call it a baseline repeat, which is also useful: it is how the noise floor gets measured. ### 2. Check the machine first Twelve clones of a React app are about 8GB of files Spotlight has never seen, and indexing them has taken this machine to a load average of 260. Before baking: ```bash ls "$BENCH_CELLS_DIR"/.metadata_never_index || echo "MISSING" ``` If it is missing, or if `ps aux | grep mds_stores` shows Spotlight busy, tell the user to run `sudo mdutil -i off <the cells directory>` (the expanded `$BENCH_CELLS_DIR`) and wait for them. Do not start a round on a machine that is already struggling. ### 3. Bake ```bash npm run compile # the driver dispatches the GUI preamble from dist/ node "$SANDBOXES_REPO"/bench/sync-templates.mjs --sizes <sizes> --ext <ext> --speckit <speckit> --sweep "<label>" ``` `--ext code` measures the extension in `COMPANION_DIR`, which is what an experiment on unreleased work needs. Use `--ext latest` only when the question is explicitly about the published build. The bake prints the three versions it recorded and fails loudly if any cell can read what it is. Report the versions; do not report the arm mapping unless asked, and never pass it to a driver or a judge. ### 4. Prep and drive ```bash node "$SANDBOXES_REPO"/bench/run-all.mjs prep --sizes <sizes> ``` Then one driver per cell, all twelve at once, following step 3 of `/bench-run-all` and the driver contract in `$SANDBOXES_REPO/bench/DRIVER.md` — **one command per step, never `/speckit-companion-auto`** — the same GUI preamble, the same settle-wait, capture for the Companion arms only, and **the cell's letter, never its arm**. Twelve drivers in parallel is fine; the round costs the slowest cell rather than the sum. Expect 20 to 30 minutes. ### 5. Measure — with nothing else running ```bash node "$SANDBOXES_REPO"/bench/run-all.mjs capture --sizes <sizes> --no-reset ``` This runs a build and two test passes in each of twelve cells. **Do not start the judges while it runs.** Stacking twelve judge agents on top of it is what took the machine down; it costs nothing to wait the ten minutes. `--no-reset` keeps the diffs so the judges have something to read. ### 6. Judge — after capture finishes, in small batches A rubric judge per cell and a comparative reviewer per size, as `/bench-capture` steps 3 and 4 describe. Run them **three or four at a time**, not twelve. Then re-run `capture` once to fold the rubric into the composite. If the machine is under load, or the user has other work to do, stop after step 5 and say so: the oracle, wall-clock and test columns are complete without the judges, and the composite can be filled in later. ### 7. Report ```bash node "$SANDBOXES_REPO"/bench/run-all.mjs compare --metric work node "$SANDBOXES_REPO"/bench/run-all.mjs compare --metric oracle node "$SANDBOXES_REPO"/bench/run-all.mjs compare --metric wall node "$SANDBOXES_REPO"/bench/run-all.mjs runs --limit 12 # the index: what ran, on which model and which build ``` Lead with what moved and whether it moved more than the spread of the sweeps before it. A change smaller than that spread has not been measured — say so plainly rather than reporting it as a result. **Read `work`, not `overall`, when comparing the stock arm to a Companion one.** `overall` gives a quarter of its weight to lifecycle capture, which stock does not have and never will, so it caps stock at 75 by construction. `work` is correctness plus rubric and every arm can reach 100. Then commit the round in `speckit-sandboxes` — `stats.jsonl`, `history.jsonl`, `runs/`, `reviews/`, `REPORT.md`, all under `bench/` — with a message naming the sweep and what it was testing. ### If something is incomplete Say which cells are missing and which columns they cost. A round with eleven of twelve cells and no rubrics is still worth pushing; a round reported as complete when it is not is worse than no round.
© alfredoperez · MIT · alfredoperez/speckit-companion