Skip to main content
command CX Codex

bench-experiment — Claude Code command for benchmark runs

bench-experiment is a Claude Code command that runs a Conduit benchmark round and compares results with earlier sweeps to measure changes across selected cells.

MIT ★ 110 updated 2026-10-19

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/commands/bench-experiment.md
Open on GitHub

What bench-experiment does

The command prepares benchmark cells, dispatches drivers, captures metrics, and compares work, oracle, and wall-clock results with earlier runs. It then records the round in speckit-sandboxes, including stats.jsonl, history.jsonl, REPORT.md, run data, and reviews.

Before baking cells, it checks for the indexing marker and active Spotlight work. If the marker is missing or Spotlight is busy, the round waits while the user runs the specified mdutil command. Judges must wait until capture finishes, then run in small batches. Running them alongside capture has previously brought the machine down.

Install it by saving the command file at .claude/commands/bench-experiment.md. The experiment description can include options for sizes, modes, extension version, spec-kit version, driver model, and GUI preamble. When comparing the stock arm with Companion, use work: overall includes lifecycle capture, which the stock arm does not have.

What it helps with

  • compare benchmark results with earlier sweeps
  • test an unreleased extension build across selected cell sizes
  • capture work, oracle, and wall-clock metrics
  • save a benchmark round and its report in speckit-sandboxes

Example prompts for the agent

  • Run a benchmark experiment called “hook anchors” across all sizes and compare it with earlier sweeps.
  • Repeat the baseline using --ext code and report the metrics.
  • Compare the Companion and stock arms using the work metric.

How to install bench-experiment

Manually: put the files in place

.claude/commands/bench-experiment.md

Call it with /bench-experiment. For all projects use ~/.claude/commands/

Questions about bench-experiment

How do I install bench-experiment in Claude Code?

Save the command file to .claude/commands/bench-experiment.md.

What does bench-experiment do?

It runs a Conduit benchmark round, compares its metrics with earlier sweeps, and records the results and report in speckit-sandboxes.

Why use work to compare the stock arm with Companion?

overall counts lifecycle capture, which the stock arm does not have. work compares correctness and rubric across the arms.

Source text

bench-experiment.md · MIT
---
allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash(node "$SANDBOXES_REPO"/bench/sync-templates.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/run-all.mjs:*), Bash(npm run compile:*), Bash(git -C:*), Bash(date:*), Bash(ps:*), Agent, Workflow, AskUserQuestion
description: Run one full benchmark round on Conduit and report it against previous sweeps
---

## Your task

Run one complete round of the benchmark and report what changed. The user says "run an experiment and bench it" and expects the whole thing without being asked for details.

The harness lives in the `bench/` folder of the sibling [`speckit-sandboxes`](https://github.com/alfredoperez/speckit-sandboxes) repo and the app it measures in [`conduit`](https://github.com/alfredoperez/conduit). `bench/README.md` there is the reference; this command is the procedure.

### Paths

Shell state does not persist between Bash calls, so start each one with this line:

```bash
. .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)"
```

It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable.

### 1. Name the sweep

`$ARGUMENTS` is what the experiment is testing, plus optional flags:

| Flag | Default | What it does |
|---|---|---|
| `--sizes easy,hard` | all four | which sizes the round covers |
| `--modes speckit,companion` | all arms | which arms to run |
| `--ext code\|latest\|<tag>` | `code` | `code` = the working tree, `latest` = the published build, or a release tag to backtrack to |
| `--speckit latest\|keep` | `latest` | pin or refresh spec-kit itself |
| `--model sonnet\|opus` | the session's | model the driver agents run on |
| `--preamble on\|off` | `off` | whether drivers are handed the GUI preamble |

**`--preamble on` is a promise the drivers have to keep.** The bake records it, nothing enforces it: a round baked `on` whose drivers were handed nothing has a column that lies, which is worse than no column. When it is on, every driver prompt starts with the output of `node "$SANDBOXES_REPO"/bench/preamble.mjs creation <speckit|companion> <specDir>` — the same text the GUI sends — and the arm decides which workflow name it takes. When it is off, drivers dispatch the command bodies alone.

Pass `--sizes`/`--modes` straight through to `sync-templates.mjs`, `prep` and `capture` — the same list at every step, or the round measures cells it never ran. Pass `--model` as the `model` option on each driver Agent. Everything else in `$ARGUMENTS` is what the experiment is testing. Turn it into a short label — "hook anchors", "living specs off", "fresh baseline". If there is nothing to name, use today's date and call it a baseline repeat, which is also useful: it is how the noise floor gets measured.

### 2. Check the machine first

Twelve clones of a React app are about 8GB of files Spotlight has never seen, and indexing them has taken this machine to a load average of 260. Before baking:

```bash
ls "$BENCH_CELLS_DIR"/.metadata_never_index || echo "MISSING"
```

If it is missing, or if `ps aux | grep mds_stores` shows Spotlight busy, tell the user to run `sudo mdutil -i off <the cells directory>` (the expanded `$BENCH_CELLS_DIR`) and wait for them. Do not start a round on a machine that is already struggling.

### 3. Bake

```bash
npm run compile                          # the driver dispatches the GUI preamble from dist/
node "$SANDBOXES_REPO"/bench/sync-templates.mjs --sizes <sizes> --ext <ext> --speckit <speckit> --sweep "<label>"
```

`--ext code` measures the extension in `COMPANION_DIR`, which is what an experiment on unreleased work needs. Use `--ext latest` only when the question is explicitly about the published build.

The bake prints the three versions it recorded and fails loudly if any cell can read what it is. Report the versions; do not report the arm mapping unless asked, and never pass it to a driver or a judge.

### 4. Prep and drive

```bash
node "$SANDBOXES_REPO"/bench/run-all.mjs prep --sizes <sizes>
```

Then one driver per cell, all twelve at once, following step 3 of `/bench-run-all` and the driver contract in `$SANDBOXES_REPO/bench/DRIVER.md` — **one command per step, never `/speckit-companion-auto`** — the same GUI preamble, the same settle-wait, capture for the Companion arms only, and **the cell's letter, never its arm**. Twelve drivers in parallel is fine; the round costs the slowest cell rather than the sum.

Expect 20 to 30 minutes.

### 5. Measure — with nothing else running

```bash
node "$SANDBOXES_REPO"/bench/run-all.mjs capture --sizes <sizes> --no-reset
```

This runs a build and two test passes in each of twelve cells. **Do not start the judges while it runs.** Stacking twelve judge agents on top of it is what took the machine down; it costs nothing to wait the ten minutes.

`--no-reset` keeps the diffs so the judges have something to read.

### 6. Judge — after capture finishes, in small batches

A rubric judge per cell and a comparative reviewer per size, as `/bench-capture` steps 3 and 4 describe. Run them **three or four at a time**, not twelve. Then re-run `capture` once to fold the rubric into the composite.

If the machine is under load, or the user has other work to do, stop after step 5 and say so: the oracle, wall-clock and test columns are complete without the judges, and the composite can be filled in later.

### 7. Report

```bash
node "$SANDBOXES_REPO"/bench/run-all.mjs compare --metric work
node "$SANDBOXES_REPO"/bench/run-all.mjs compare --metric oracle
node "$SANDBOXES_REPO"/bench/run-all.mjs compare --metric wall
node "$SANDBOXES_REPO"/bench/run-all.mjs runs --limit 12     # the index: what ran, on which model and which build
```

Lead with what moved and whether it moved more than the spread of the sweeps before it. A change smaller than that spread has not been measured — say so plainly rather than reporting it as a result.

**Read `work`, not `overall`, when comparing the stock arm to a Companion one.** `overall` gives a quarter of its weight to lifecycle capture, which stock does not have and never will, so it caps stock at 75 by construction. `work` is correctness plus rubric and every arm can reach 100.

Then commit the round in `speckit-sandboxes` — `stats.jsonl`, `history.jsonl`, `runs/`, `reviews/`, `REPORT.md`, all under `bench/` — with a message naming the sweep and what it was testing.

### If something is incomplete

Say which cells are missing and which columns they cost. A round with eleven of twelve cells and no rubrics is still worth pushing; a round reported as complete when it is not is worse than no round.

© alfredoperez · MIT · alfredoperez/speckit-companion

Similar slash commands

All slash commands for Testing and QA →

From the same set83

The whole set →
bench-replay The bench-replay command for Claude Code and Codex replays one pipeline step from a saved stage and grades the effect of a change. CX Codex +1 ★ 110repo stars speckit.tasks The speckit.tasks prompt turns feature design documents into a dependency-ordered tasks.md, grouped by user story. Use it to prepare an implementation plan from an existing spec. CX Codex GP ChatGPT ★ 110repo stars changelog-entry changelog-entry is a Claude Code command that writes a changelog entry from a PR or current diff and chooses its file and format. Run it with a PR number or no argument. CX Codex +1 ★ 110repo stars eval-speckit-extension eval-speckit-extension is a Claude Code skill that tests speckit-extension against a completed spec, verifying lifecycle hooks and .spec-context.json timestamps. CX Codex +1 ★ 110repo stars speckit.analyze speckit.analyze runs a read-only pass over spec.md, plan.md and tasks.md, then prints a report on contradictions, duplicates and requirements that no task covers. CX Codex GP ChatGPT ★ 110repo stars speckit.companion.living-sync speckit.companion.living-sync updates living specs to match current working-tree changes in a single pass. Modifies existing spec files without regenerating them from scratch. CX Codex +1 ★ 110repo stars speckit.implement The speckit.implement prompt turns the plan in tasks.md into an ordered implementation run. It is listed for Claude Code and Codex. CX Codex GP ChatGPT ★ 110repo stars publish-all publish-all releases all four project components at once: the spec-kit extension, VS Code extension, Claude Code mod, and Copilot canvas. Runs only if changes are ready and pass QA checks. CX Codex +1 ★ 110repo stars

Project sponsors

Companies that keep this analytics open