bench-replay — Claude Code and Codex command: replay one pipeline step
The bench-replay command for Claude Code and Codex replays one pipeline step from a saved stage and grades the effect of a change.
MIT ★ 110 updated 2026-10-19
.claude/commands/bench-replay.mdWhat bench-replay does
Works in Claude Code and Codex. It answers "does X help", not "is the pipeline good today" — the latter is what a full round via /bench-experiment is for.
You name the change and the step it touches: implement, tasks, plan, or specify. The command restores the stage before that step, runs the shipped behavior and one or two variants, then grades each with the exam and checks regression cases. Each result lands on a branch replay-<label> for diffing.
A replay costs about $8 and takes ten minutes; a round is about $35 and forty. A run that finishes in under two minutes with zero workers did not run — read runs-meta/replay-<cell>-<stage>-<step>-<label>.json before trusting a low score.
What it helps with
- test whether one change helps without running a full round
- replay a single step from a saved stage while holding the rest fixed
- get an exam score for each variant of a change
- trace a failing variant down to the line of code
- compare variants by time, cost, and workers
Example prompts for the agent
- Check whether turning fan-out off helps the implement step
- Replay the plan step from post-specify for gentle-ai's classifier
- Compare asis against the new check node on tasks and break down scores per variant
How to install bench-replay
Manually: put the files in place
.claude/commands/bench-replay.mdCall it with /bench-replay. For all projects use ~/.claude/commands/
Manually: put the files in place
~/.codex/prompts/bench-replay.mdCall it with /prompts:bench-replay. Codex marks this format deprecated in favour of skills
Manually: put the files in place
.github/prompts/bench-replay.prompt.mdManually: put the files in place
.gemini/commands/bench-replay.tomlGemini stores commands as TOML: the prompt text goes into the prompt field
Questions about bench-replay
How do I install bench-replay in Claude Code?
Copy .claude/commands/bench-replay.md from the alfredoperez/speckit-companion repository into the .claude/commands folder of your project.
What does bench-replay do?
It replays one pipeline step (implement, tasks, plan, or specify) from a saved stage, with and without the change, and grades each variant. The rest of the environment stays identical.
How is a replay different from a full round?
A replay costs about $8 and takes ten minutes; a round is about $35 and forty minutes. Replays answer "does X help", rounds answer "is the pipeline good today". Replay results never land in history.jsonl.
Source text
bench-replay.md · MIT
---
allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash("$SANDBOXES_REPO"/bench/replay.sh:*), Bash("$SANDBOXES_REPO"/bench/stage-from-capture.sh:*), Bash(node "$SANDBOXES_REPO"/bench/grade-one.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/regressions/read-case.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/sync-templates.mjs:*), Bash(node -e:*), Bash(git -C:*), Bash(ls:*), Bash(cat:*), Bash(python3:*), AskUserQuestion
description: Change one thing and replay one pipeline step from a staged cell, then grade it
---
## Your task
Answer a single-step question without running a round. The user names a change ("fan-out off", "the new check node", "gentle-ai's classifier") and which step it affects; you replay that step from a staged state, as many ways as the question needs, and grade each. Everything else is held byte-identical, so a difference in the result is the change and nothing else.
This is for "does X help", never for "is the pipeline good today". A round answers the second. A replay costs about $8 and ten minutes; a round costs about $35 and forty. The harness `README.md` (`bench/README.md` in `speckit-sandboxes`) under **Stages** is the reference; this command is the procedure.
### Paths
Shell state does not persist between Bash calls, so start each one with this line:
```bash
. .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)"
```
It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable.
### 1. Name the question and the variants
`$ARGUMENTS` is the change. Turn it into:
- the **step** it affects: `implement`, `tasks`, `plan`, or `specify`
- the **stage** to start from: the one before that step (`post-tasks` for implement, `post-plan` for tasks, `post-specify` for plan, the baseline for specify)
- the **variants**, each with a one-word label: the shipped behaviour is always one of them and is always labelled `asis`
Two variants is the normal shape. Three or more only when the question genuinely has that many arms.
### 2. Pick the cell and check the build
Pick the arm the question is about. Cells are blind on disk; identify one by what is installed rather than by letter:
```bash
for d in "$BENCH_CELLS_DIR"/conduit-*/; do n=$(basename "$d"); c=$([ -d "$d/.specify/extensions/companion" ] && echo companion || echo stock); l=$([ -f "$d/living-specs.yml" ] && echo "+living" || echo ""); echo "$n $c$l"; done
```
A variant that changes the product needs the pinned build to carry it: `~/dev/GitHub/speckit-companion.worktrees/bench-main` is what a cell installs from. If the change is on `main` and not in the worktree, fast-forward it, then refresh the cell in place rather than re-baking:
```bash
git -C ~/dev/GitHub/speckit-companion.worktrees/bench-main merge --ff-only main
node -e "import('$SANDBOXES_REPO/bench/sync-templates.mjs').then(m=>m.installCompanion('$BENCH_CELLS_DIR/<cell>','companion','code'))"
node -e "import('$SANDBOXES_REPO/bench/lib.mjs').then(m=>m.gitCommitCellBaseline('$BENCH_CELLS_DIR/<cell>'))"
```
The second line re-tags the baseline so the stage you build next starts from the new build. Skip both if the variant is only a prompt passed to `replay.sh`.
### 3. Get a stage
Check for one first:
```bash
git -C "$BENCH_CELLS_DIR"/<cell> tag | grep ^stage/
```
If the stage you need is missing, rebuild it from a captured run. Every round since the clean sweep leaves stages behind, and every captured run under `runs/conduit/<runId>/` can become one:
```bash
"$SANDBOXES_REPO"/bench/stage-from-capture.sh <runId> <cell> <stage>
```
It winds the record back — unticks the task boxes, resets the status, trims the history — because a capture is of a finished run and a replayed step would otherwise read it as work already done. Confirm the output says `source changes: 0`.
### 4. Replay each variant
One call per variant. A variant that is only an instruction goes as the fifth argument; a variant that is a product change was installed in step 2:
```bash
"$SANDBOXES_REPO"/bench/replay.sh <cell> <stage> <step> asis
"$SANDBOXES_REPO"/bench/replay.sh <cell> <stage> <step> <label> "<extra instruction>"
```
Run them one after another, never at once: the machine and the workers are what is being measured. Each replay prints its time, tokens, workers and cost, then the exam score, then any regression case the cell still trips. The result stays on branch `replay-<label>` in the cell for diffing.
**A replay that finishes in under two minutes with zero workers did not run.** Read `$SANDBOXES_REPO/bench/runs-meta/replay-<cell>-<stage>-<step>-<label>.json` before believing a low score. Twice today a harness bug read as a catastrophic result.
### 5. Report
A table, one row per variant: exam, workers, cost, time, and the regression cases. Then the answer to the question in one or two sentences, and the strongest thing that disagrees with it.
Then, for every variant that scored below the best one, **chase the failure to the line** before reporting it. The per-test detail is in `.test-out/<cell>.acceptance.json`, the code is on the replay branch. Two variants failing the same tests means one cause. A score with no named cause is a number, not a finding.
Replays never land in `history.jsonl`. If the answer changes what ships, the confirming step is a round via `/bench-experiment`, not another replay.
### What not to do
- Do not compare a replay row against a round row. Different starting states.
- Do not reuse a cell's stage after refreshing its build without re-staging. The old stage carries the old commands.
- Do not stage by hand with `git`. `stage-from-capture.sh` and `drive-steps.sh` are the two writers, and both wind the record back.
© alfredoperez · MIT · alfredoperez/speckit-companion