Skip to main content
command CX Codex

bench-replay — Claude Code and Codex command: replay one pipeline step

The bench-replay command for Claude Code and Codex replays one pipeline step from a saved stage and grades the effect of a change.

MIT ★ 110 updated 2026-10-19

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/commands/bench-replay.md
Open on GitHub

What bench-replay does

Works in Claude Code and Codex. It answers "does X help", not "is the pipeline good today" — the latter is what a full round via /bench-experiment is for.

You name the change and the step it touches: implement, tasks, plan, or specify. The command restores the stage before that step, runs the shipped behavior and one or two variants, then grades each with the exam and checks regression cases. Each result lands on a branch replay-<label> for diffing.

A replay costs about $8 and takes ten minutes; a round is about $35 and forty. A run that finishes in under two minutes with zero workers did not run — read runs-meta/replay-<cell>-<stage>-<step>-<label>.json before trusting a low score.

What it helps with

  • test whether one change helps without running a full round
  • replay a single step from a saved stage while holding the rest fixed
  • get an exam score for each variant of a change
  • trace a failing variant down to the line of code
  • compare variants by time, cost, and workers

Example prompts for the agent

  • Check whether turning fan-out off helps the implement step
  • Replay the plan step from post-specify for gentle-ai's classifier
  • Compare asis against the new check node on tasks and break down scores per variant

How to install bench-replay

Manually: put the files in place

.claude/commands/bench-replay.md

Call it with /bench-replay. For all projects use ~/.claude/commands/

Questions about bench-replay

How do I install bench-replay in Claude Code?

Copy .claude/commands/bench-replay.md from the alfredoperez/speckit-companion repository into the .claude/commands folder of your project.

What does bench-replay do?

It replays one pipeline step (implement, tasks, plan, or specify) from a saved stage, with and without the change, and grades each variant. The rest of the environment stays identical.

How is a replay different from a full round?

A replay costs about $8 and takes ten minutes; a round is about $35 and forty minutes. Replays answer "does X help", rounds answer "is the pipeline good today". Replay results never land in history.jsonl.

Source text

bench-replay.md · MIT
---
allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash("$SANDBOXES_REPO"/bench/replay.sh:*), Bash("$SANDBOXES_REPO"/bench/stage-from-capture.sh:*), Bash(node "$SANDBOXES_REPO"/bench/grade-one.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/regressions/read-case.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/sync-templates.mjs:*), Bash(node -e:*), Bash(git -C:*), Bash(ls:*), Bash(cat:*), Bash(python3:*), AskUserQuestion
description: Change one thing and replay one pipeline step from a staged cell, then grade it
---

## Your task

Answer a single-step question without running a round. The user names a change ("fan-out off", "the new check node", "gentle-ai's classifier") and which step it affects; you replay that step from a staged state, as many ways as the question needs, and grade each. Everything else is held byte-identical, so a difference in the result is the change and nothing else.

This is for "does X help", never for "is the pipeline good today". A round answers the second. A replay costs about $8 and ten minutes; a round costs about $35 and forty. The harness `README.md` (`bench/README.md` in `speckit-sandboxes`) under **Stages** is the reference; this command is the procedure.

### Paths

Shell state does not persist between Bash calls, so start each one with this line:

```bash
. .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)"
```

It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable.

### 1. Name the question and the variants

`$ARGUMENTS` is the change. Turn it into:

- the **step** it affects: `implement`, `tasks`, `plan`, or `specify`
- the **stage** to start from: the one before that step (`post-tasks` for implement, `post-plan` for tasks, `post-specify` for plan, the baseline for specify)
- the **variants**, each with a one-word label: the shipped behaviour is always one of them and is always labelled `asis`

Two variants is the normal shape. Three or more only when the question genuinely has that many arms.

### 2. Pick the cell and check the build

Pick the arm the question is about. Cells are blind on disk; identify one by what is installed rather than by letter:

```bash
for d in "$BENCH_CELLS_DIR"/conduit-*/; do n=$(basename "$d"); c=$([ -d "$d/.specify/extensions/companion" ] && echo companion || echo stock); l=$([ -f "$d/living-specs.yml" ] && echo "+living" || echo ""); echo "$n $c$l"; done
```

A variant that changes the product needs the pinned build to carry it: `~/dev/GitHub/speckit-companion.worktrees/bench-main` is what a cell installs from. If the change is on `main` and not in the worktree, fast-forward it, then refresh the cell in place rather than re-baking:

```bash
git -C ~/dev/GitHub/speckit-companion.worktrees/bench-main merge --ff-only main
node -e "import('$SANDBOXES_REPO/bench/sync-templates.mjs').then(m=>m.installCompanion('$BENCH_CELLS_DIR/<cell>','companion','code'))"
node -e "import('$SANDBOXES_REPO/bench/lib.mjs').then(m=>m.gitCommitCellBaseline('$BENCH_CELLS_DIR/<cell>'))"
```

The second line re-tags the baseline so the stage you build next starts from the new build. Skip both if the variant is only a prompt passed to `replay.sh`.

### 3. Get a stage

Check for one first:

```bash
git -C "$BENCH_CELLS_DIR"/<cell> tag | grep ^stage/
```

If the stage you need is missing, rebuild it from a captured run. Every round since the clean sweep leaves stages behind, and every captured run under `runs/conduit/<runId>/` can become one:

```bash
"$SANDBOXES_REPO"/bench/stage-from-capture.sh <runId> <cell> <stage>
```

It winds the record back — unticks the task boxes, resets the status, trims the history — because a capture is of a finished run and a replayed step would otherwise read it as work already done. Confirm the output says `source changes: 0`.

### 4. Replay each variant

One call per variant. A variant that is only an instruction goes as the fifth argument; a variant that is a product change was installed in step 2:

```bash
"$SANDBOXES_REPO"/bench/replay.sh <cell> <stage> <step> asis
"$SANDBOXES_REPO"/bench/replay.sh <cell> <stage> <step> <label> "<extra instruction>"
```

Run them one after another, never at once: the machine and the workers are what is being measured. Each replay prints its time, tokens, workers and cost, then the exam score, then any regression case the cell still trips. The result stays on branch `replay-<label>` in the cell for diffing.

**A replay that finishes in under two minutes with zero workers did not run.** Read `$SANDBOXES_REPO/bench/runs-meta/replay-<cell>-<stage>-<step>-<label>.json` before believing a low score. Twice today a harness bug read as a catastrophic result.

### 5. Report

A table, one row per variant: exam, workers, cost, time, and the regression cases. Then the answer to the question in one or two sentences, and the strongest thing that disagrees with it.

Then, for every variant that scored below the best one, **chase the failure to the line** before reporting it. The per-test detail is in `.test-out/<cell>.acceptance.json`, the code is on the replay branch. Two variants failing the same tests means one cause. A score with no named cause is a number, not a finding.

Replays never land in `history.jsonl`. If the answer changes what ships, the confirming step is a round via `/bench-experiment`, not another replay.

### What not to do

- Do not compare a replay row against a round row. Different starting states.
- Do not reuse a cell's stage after refreshing its build without re-staging. The old stage carries the old commands.
- Do not stage by hand with `git`. `stage-from-capture.sh` and `drive-steps.sh` are the two writers, and both wind the record back.

© alfredoperez · MIT · alfredoperez/speckit-companion

Similar slash commands

All slash commands for Agent setup: memory, context, orchestration →

From the same set83

The whole set →
speckit.companion.mark-complete speckit.companion.mark-complete marks the active spec as completed via write-context.py. Handy when you want to close a spec by hand instead of running a full converge. CX Codex +1 ★ 110repo stars speckit.constitution speckit.constitution updates the project constitution template in .specify/memory/constitution.md and aligns dependent specification templates. CX Codex GP ChatGPT ★ 110repo stars bench-run-all The bench-run-all command runs one benchmark size across three cells, drives the agent steps, then judges and captures the results. It is built for Speckit pipeline comparisons. CX Codex +1 ★ 110repo stars publish-canvas publish-canvas ships the Copilot app canvas: version bump, tests, the canvas-v<X.Y.Z> tag, then the github/awesome-copilot listing update. CX Codex +1 ★ 110repo stars speckit.tasks The speckit.tasks prompt turns feature design documents into a dependency-ordered tasks.md, grouped by user story. Use it to prepare an implementation plan from an existing spec. CX Codex GP ChatGPT ★ 110repo stars changelog-entry changelog-entry is a Claude Code command that writes a changelog entry from a PR or current diff and chooses its file and format. Run it with a PR number or no argument. CX Codex +1 ★ 110repo stars eval-speckit-extension eval-speckit-extension is a Claude Code skill that tests speckit-extension against a completed spec, verifying lifecycle hooks and .spec-context.json timestamps. CX Codex +1 ★ 110repo stars bench-experiment bench-experiment is a Claude Code command that runs a Conduit benchmark round and compares results with earlier sweeps to measure changes across selected cells. CX Codex +1 ★ 110repo stars

Project sponsors

Companies that keep this analytics open