bench-run-all — Claude Code command: benchmark run and setup
The bench-run-all command runs one benchmark size across three cells, drives the agent steps, then judges and captures the results. It is built for Speckit pipeline comparisons.
MIT ★ 110 updated 2026-10-19
.claude/commands/bench-run-all.mdWhat bench-run-all does
bench-run-all automates a round for one size. It checks the cells, prepares them, starts three drivers in parallel, and takes each cell through specify, plan, tasks, and implement. After the runs, it launches a rubric judge and a cross-solution reviewer, captures the results, and prints a three-column report with Overall rows and version data. Supported size arguments are easy, medium, hard, and oversized.
Each shell call starts by loading .claude/sandboxes-env.sh; the workflow then uses run-all.mjs plus the paths under $SANDBOXES_REPO and $BENCH_CELLS_DIR. Drivers may run the project build and test suite. They must not run git commands, because changing a cell baseline would invalidate the comparison. The benchmark compares relative results with capture overhead separated from the measurement; it does not reproduce a person's absolute wall-clock time.
For Claude Code, install the command as .claude/commands/bench-run-all.md.
What it helps with
- check whether the three benchmark cells are ready
- run one size through specify, plan, tasks, and implement
- drive all three cells in parallel
- judge each result and compare the solutions
- print the Overall rows with version data
Example prompts for the agent
- Run
/bench-run-all easyand print the final comparison table. - Execute
/bench-run-all mediumwhile allowing the project test suite to run. - Repeat the benchmark with
/bench-run-all oversized.
How to install bench-run-all
Manually: put the files in place
.claude/commands/bench-run-all.mdCall it with /bench-run-all. For all projects use ~/.claude/commands/
Manually: put the files in place
~/.codex/prompts/bench-run-all.mdCall it with /prompts:bench-run-all. Codex marks this format deprecated in favour of skills
Manually: put the files in place
.github/prompts/bench-run-all.prompt.mdManually: put the files in place
.gemini/commands/bench-run-all.tomlGemini stores commands as TOML: the prompt text goes into the prompt field
Questions about bench-run-all
How do I install bench-run-all in Claude Code?
Create .claude/commands/bench-run-all.md and place the command content in that file.
What does bench-run-all do?
It prepares three cells for one size, assigns drivers to the Speckit steps, then runs judging, cross-solution review, capture, and reporting.
Why must bench-run-all drivers avoid git commands?
Branching and commits belong to the pipeline. A git command from a driver can rewrite a cell baseline and distort the benchmark result.
Source text
bench-run-all.md · MIT
--- allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash(node "$SANDBOXES_REPO"/bench/run-all.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/sync-templates.mjs:*), Bash(date:*), Bash(git -C:*), Agent, Workflow, AskUserQuestion description: Agent-driven bench round for one size — drive the three cells, judge, capture --- ## Your task The automated version of `/bench-prep` → run it → `/bench-capture`, for one size. Driver agents work the cells instead of a person. Size from `$ARGUMENTS` (`easy`/`medium`/`hard`/`oversized`). > Faithful dispatch: a driver mimics the GUI, it does not follow raw command bodies. > > **Never `/speckit-companion-auto`.** It is a different product path — it decides its own shape and folds on its own judgement — and a round driven by it is measuring auto, not the pipeline a GUI user drives. A round that used it also loaded no living specs at all in one measured cell, while the per-step run of the same feature in the same cell loaded three and caught a planted contradiction. The full driver contract is `$SANDBOXES_REPO/bench/DRIVER.md`. The bench is a trustworthy **relative** comparator with capture overhead isolated; it does not reproduce a human's absolute wall clock. ### Paths Shell state does not persist between Bash calls, so start each one with this line: ```bash . .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)" ``` It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable. ### 1. Check the cells `node "$SANDBOXES_REPO"/bench/run-all.mjs --dry-run` — if any cell is missing, run `/bench-sync`. Note the three versions it prints; they belong in the final report. ### 2. Prep `node "$SANDBOXES_REPO"/bench/run-all.mjs prep --sizes <size>` — resets the cells and writes their run markers into the harness. ### 3. Drive the three cells (Workflow, parallel) One driver per cell (`parallel` of 3). Each works only in `$BENCH_CELLS_DIR/conduit-<size>-<letter>`, stamps `startedAt`/`finishedAt` into `$SANDBOXES_REPO/bench/runs-meta/conduit-<size>-<letter>.json` (via `date -u`), and runs **specify → plan → tasks → implement** the GUI-faithful way (see `$SANDBOXES_REPO/bench/driver.mjs`): - For EACH step, prepend the **same** GUI preamble every arm gets — `buildStepPreamble(step, specDir)` from `driver.mjs`, which imports the real renderer from `dist/ai-providers/promptPreamble.js` so it cannot drift — then dispatch the step's command. - **stock arm** → stock `/speckit.*` command bodies. No capture script; stock is blind by design. - **Companion arms** → `/speckit.companion.*` command bodies, **and nothing on top of them**. The bodies carry every capture call a run needs. A driver that also runs `cap.mjs <step> start` writes the start stamp twice — the second is a documented no-op that still costs a round-trip on every step, and it inflated wave A's call count by four. The count that matters comes from the writer's own `.trace.jsonl`, so the driver has no bookkeeping of its own to do. - After dispatching a step, **wait for it to settle** — `waitForSettle(cellDir, step)` polls `.spec-context.json` until the step's completed status **or any later one** appears. It returns `folded: true` when the status overshot, which happens for two shipped reasons: the fast path folds specify/plan/tasks onto `ready-to-implement`, and mark-complete takes implement to `completed`. **A folded step is already done — never re-dispatch it, and never steer the size verdict to make steps settle one at a time.** Right-sizing is the feature under measurement; a driver that disables it produces numbers that look valid and are not. - Do not time the capture calls. The harness counts them from `.trace.jsonl` and the timing itself is under the noise floor. The feature prompt is `$SANDBOXES_REPO/bench/prompts/conduit/<size>.md`, the text between the `---` rules. **A driver may run the app's own build and test suite, and should.** That is what a person does, and forbidding it measured something nobody does: on the first Conduit round one arm wrote twelve tests and eleven failed on a single convention slip it had no way to see, because it was not allowed to run them. The acceptance oracle is injected only at grading time and removed afterwards, so a run cannot reach it from inside the cell — there is nothing to protect by keeping the suite closed. The one hard rule for a driver is **no git commands**. Branching and committing belong to the pipeline, and a driver reaching for git is how a cell's baseline gets rewritten under the harness. **Tell a driver its letter, never its arm.** A driver that knows it is "the Companion arm" is no longer measuring anything. ### 4. Judge + capture Spawn a rubric judge per cell and one cross-solution comparative reviewer (steps 3 and 4 of `/bench-capture`), then: `node "$SANDBOXES_REPO"/bench/run-all.mjs capture --sizes <size>` ### 5. Report Print the size's three-column table with the Overall rows and the version line. Repeat for the other sizes to cover the matrix.
© alfredoperez · MIT · alfredoperez/speckit-companion