Skip to main content
command CX Codex

bench-run-all — Claude Code command: benchmark run and setup

The bench-run-all command runs one benchmark size across three cells, drives the agent steps, then judges and captures the results. It is built for Speckit pipeline comparisons.

MIT ★ 110 updated 2026-10-19

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/commands/bench-run-all.md
Open on GitHub

What bench-run-all does

bench-run-all automates a round for one size. It checks the cells, prepares them, starts three drivers in parallel, and takes each cell through specify, plan, tasks, and implement. After the runs, it launches a rubric judge and a cross-solution reviewer, captures the results, and prints a three-column report with Overall rows and version data. Supported size arguments are easy, medium, hard, and oversized.

Each shell call starts by loading .claude/sandboxes-env.sh; the workflow then uses run-all.mjs plus the paths under $SANDBOXES_REPO and $BENCH_CELLS_DIR. Drivers may run the project build and test suite. They must not run git commands, because changing a cell baseline would invalidate the comparison. The benchmark compares relative results with capture overhead separated from the measurement; it does not reproduce a person's absolute wall-clock time.

For Claude Code, install the command as .claude/commands/bench-run-all.md.

What it helps with

  • check whether the three benchmark cells are ready
  • run one size through specify, plan, tasks, and implement
  • drive all three cells in parallel
  • judge each result and compare the solutions
  • print the Overall rows with version data

Example prompts for the agent

  • Run /bench-run-all easy and print the final comparison table.
  • Execute /bench-run-all medium while allowing the project test suite to run.
  • Repeat the benchmark with /bench-run-all oversized.

How to install bench-run-all

Manually: put the files in place

.claude/commands/bench-run-all.md

Call it with /bench-run-all. For all projects use ~/.claude/commands/

Questions about bench-run-all

How do I install bench-run-all in Claude Code?

Create .claude/commands/bench-run-all.md and place the command content in that file.

What does bench-run-all do?

It prepares three cells for one size, assigns drivers to the Speckit steps, then runs judging, cross-solution review, capture, and reporting.

Why must bench-run-all drivers avoid git commands?

Branching and commits belong to the pipeline. A git command from a driver can rewrite a cell baseline and distort the benchmark result.

Source text

bench-run-all.md · MIT
---
allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash(node "$SANDBOXES_REPO"/bench/run-all.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/sync-templates.mjs:*), Bash(date:*), Bash(git -C:*), Agent, Workflow, AskUserQuestion
description: Agent-driven bench round for one size — drive the three cells, judge, capture
---

## Your task

The automated version of `/bench-prep` → run it → `/bench-capture`, for one size. Driver agents work the cells instead of a person.

Size from `$ARGUMENTS` (`easy`/`medium`/`hard`/`oversized`).

> Faithful dispatch: a driver mimics the GUI, it does not follow raw command bodies.
>
> **Never `/speckit-companion-auto`.** It is a different product path — it decides its own shape and folds on its own judgement — and a round driven by it is measuring auto, not the pipeline a GUI user drives. A round that used it also loaded no living specs at all in one measured cell, while the per-step run of the same feature in the same cell loaded three and caught a planted contradiction. The full driver contract is `$SANDBOXES_REPO/bench/DRIVER.md`. The bench is a trustworthy **relative** comparator with capture overhead isolated; it does not reproduce a human's absolute wall clock.

### Paths

Shell state does not persist between Bash calls, so start each one with this line:

```bash
. .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)"
```

It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable.

### 1. Check the cells

`node "$SANDBOXES_REPO"/bench/run-all.mjs --dry-run` — if any cell is missing, run `/bench-sync`. Note the three versions it prints; they belong in the final report.

### 2. Prep

`node "$SANDBOXES_REPO"/bench/run-all.mjs prep --sizes <size>` — resets the cells and writes their run markers into the harness.

### 3. Drive the three cells (Workflow, parallel)

One driver per cell (`parallel` of 3). Each works only in `$BENCH_CELLS_DIR/conduit-<size>-<letter>`, stamps `startedAt`/`finishedAt` into `$SANDBOXES_REPO/bench/runs-meta/conduit-<size>-<letter>.json` (via `date -u`), and runs **specify → plan → tasks → implement** the GUI-faithful way (see `$SANDBOXES_REPO/bench/driver.mjs`):

- For EACH step, prepend the **same** GUI preamble every arm gets — `buildStepPreamble(step, specDir)` from `driver.mjs`, which imports the real renderer from `dist/ai-providers/promptPreamble.js` so it cannot drift — then dispatch the step's command.
  - **stock arm** → stock `/speckit.*` command bodies. No capture script; stock is blind by design.
  - **Companion arms** → `/speckit.companion.*` command bodies, **and nothing on top of them**. The bodies carry every capture call a run needs. A driver that also runs `cap.mjs <step> start` writes the start stamp twice — the second is a documented no-op that still costs a round-trip on every step, and it inflated wave A's call count by four. The count that matters comes from the writer's own `.trace.jsonl`, so the driver has no bookkeeping of its own to do.
- After dispatching a step, **wait for it to settle** — `waitForSettle(cellDir, step)` polls `.spec-context.json` until the step's completed status **or any later one** appears. It returns `folded: true` when the status overshot, which happens for two shipped reasons: the fast path folds specify/plan/tasks onto `ready-to-implement`, and mark-complete takes implement to `completed`. **A folded step is already done — never re-dispatch it, and never steer the size verdict to make steps settle one at a time.** Right-sizing is the feature under measurement; a driver that disables it produces numbers that look valid and are not.
- Do not time the capture calls. The harness counts them from `.trace.jsonl` and the timing itself is under the noise floor.

The feature prompt is `$SANDBOXES_REPO/bench/prompts/conduit/<size>.md`, the text between the `---` rules.

**A driver may run the app's own build and test suite, and should.** That is what a person does, and forbidding it measured something nobody does: on the first Conduit round one arm wrote twelve tests and eleven failed on a single convention slip it had no way to see, because it was not allowed to run them. The acceptance oracle is injected only at grading time and removed afterwards, so a run cannot reach it from inside the cell — there is nothing to protect by keeping the suite closed.

The one hard rule for a driver is **no git commands**. Branching and committing belong to the pipeline, and a driver reaching for git is how a cell's baseline gets rewritten under the harness.

**Tell a driver its letter, never its arm.** A driver that knows it is "the Companion arm" is no longer measuring anything.

### 4. Judge + capture

Spawn a rubric judge per cell and one cross-solution comparative reviewer (steps 3 and 4 of `/bench-capture`), then:

`node "$SANDBOXES_REPO"/bench/run-all.mjs capture --sizes <size>`

### 5. Report

Print the size's three-column table with the Overall rows and the version line. Repeat for the other sizes to cover the matrix.

© alfredoperez · MIT · alfredoperez/speckit-companion

Similar slash commands

All slash commands for Agent setup: memory, context, orchestration →

From the same set83

The whole set →
speckit.companion.living-move speckit.companion.living-move relocates Speckit living specs between central storage and colocated code folders while updating the project registry. CX Codex +1 ★ 110repo stars publish-mod publish-mod releases the Claude Code mod through the repository marketplace and prepares the Anthropic directory handoff for the user. CX Codex +1 ★ 110repo stars speckit.companion.mark-complete speckit.companion.mark-complete marks the active spec as completed via write-context.py. Handy when you want to close a spec by hand instead of running a full converge. CX Codex +1 ★ 110repo stars speckit.constitution speckit.constitution updates the project constitution template in .specify/memory/constitution.md and aligns dependent specification templates. CX Codex GP ChatGPT ★ 110repo stars publish-canvas publish-canvas ships the Copilot app canvas: version bump, tests, the canvas-v<X.Y.Z> tag, then the github/awesome-copilot listing update. CX Codex +1 ★ 110repo stars bench-replay The bench-replay command for Claude Code and Codex replays one pipeline step from a saved stage and grades the effect of a change. CX Codex +1 ★ 110repo stars speckit.tasks The speckit.tasks prompt turns feature design documents into a dependency-ordered tasks.md, grouped by user story. Use it to prepare an implementation plan from an existing spec. CX Codex GP ChatGPT ★ 110repo stars changelog-entry changelog-entry is a Claude Code command that writes a changelog entry from a PR or current diff and chooses its file and format. Run it with a PR number or no argument. CX Codex +1 ★ 110repo stars

Project sponsors

Companies that keep this analytics open