Skip to main content
command CX Codex

bench-capture — Claude Code and Codex command

bench-capture — a command for Claude Code and Codex that captures benchmark scores after a size runs through three cells, appends them to history, regenerates the report, and resets the cells for the next round.

MIT ★ 110 updated 2026-10-19

Install into Claude Code · Manually: put the files in place All methods and platforms ↓
.claude/commands/bench-capture.md
Open on GitHub

What bench-capture does

The workflow has six stages: first, resolve the size (easy, medium, hard, or oversized), then fetch which letter maps to which arm and which versions are being measured. Three independent agents rate each cell on readability, adherence to conventions, and scope — each judge sees only a letter, never which arm they are reviewing. A fourth agent then compares all three solutions head-to-head, flags structural differences and suspected bugs, and writes the verdict into an accumulating report that preserves all prior runs.

The final stage runs the build, full test suite, convention checks, computes the composite score, and appends a row to the measurement history. Before measuring, the diffable cells are read so the comparative reviewer can examine them; after measurement, all three cells are automatically reset.

Requires environment variables from .claude/sandboxes-env.sh and access to the bench folders. Needs Node.js and git in PATH. Always expand variable paths before handing them to agents — they receive absolute paths only.

What it helps with

  • rate three different solutions against the same rubric without bias
  • compare architectural approaches and detect bugs before performance measurement
  • merge all judge scores into a single report while preserving the full run history
  • reset benchmark cells to a clean state for the next round

Example prompts for the agent

  • Score the easy size after three cells: run bench-capture and select easy
  • Record judge ratings and run tests for medium: invoke bench-capture, pick medium, wait for all judges and the report
  • Run a full benchmark cycle for hard: use bench-capture, choose hard, wait for comparative review and test output

How to install bench-capture

Manually: put the files in place

.claude/commands/bench-capture.md

Call it with /bench-capture. For all projects use ~/.claude/commands/

Questions about bench-capture

How do I install bench-capture in Claude Code?

The command file is at .claude/commands/bench-capture.md. Copy it into your Claude Code commands folder or use the setup dialog.

What does bench-capture do?

It captures benchmark results: spawns three independent judges to score each cell, runs a comparative review of all solutions, collects ratings, executes the full test suite, and resets the cells for the next round.

Why don't judges learn which solution they are reviewing?

To keep scoring impartial. A judge told they are looking at the Companion arm may unconsciously skew the score. Each receives only a letter and instructions.

Source text

bench-capture.md · MIT
---
allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash(node "$SANDBOXES_REPO"/bench/run-all.mjs:*), Bash(git -C:*), Agent, Workflow, AskUserQuestion
description: Score the three bench cells for one size, record, report, and reset
---

## Your task

After a size has been run through its three cells, capture the evals, append to history, regenerate the report, and reset the cells for the next round.

### Paths

Shell state does not persist between Bash calls, so start each one with this line:

```bash
. .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)"
```

It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable.

### 1. Resolve the size

From `$ARGUMENTS` — `easy` | `medium` | `hard` | `oversized`. If missing, ask with **AskUserQuestion**.

### 2. Read the cell mapping

```bash
node "$SANDBOXES_REPO"/bench/run-all.mjs --dry-run
```

It prints which letter carries which arm, plus the spec-kit, extension and Companion versions this round was measured with. You need the mapping to attribute the reviews; the judges below must **not** be told which arm they are looking at.

### 3. Independent rubric judges (before measuring — they read the diff)

Spawn one **judge agent per cell** (a `parallel()` Workflow of 3). Each judge — which did NOT write the code — does exactly:

- Read the diff: `git -C "$BENCH_CELLS_DIR"/conduit-<size>-<letter> diff bench-baseline`.
- Read the requirements at `$SANDBOXES_REPO/bench/prompts/conduit/<size>.md` and the conventions in the cell's `CLAUDE.md`.
- Score 1–5: `readability`, `conventions`, `scope`. Judge the layering rules the app actually documents; do not penalize a rule the size had no occasion to exercise.
- Write `quality = {readability, conventions, scope, justification}` into `$SANDBOXES_REPO/bench/runs-meta/conduit-<size>-<letter>.json` (MERGE — keep `runId`/`size`/`mode`).

Give each judge the **letter only**. A judge told it is looking at "the Companion arm" is no longer independent.

### 4. Comparative reviewer (one agent, cross-solution)

The per-cell judges score each solution in isolation, and tests cannot fully pin correctness. Spawn **one** reviewer that sees all three at once. **Run it before step 5** — measuring resets the cells and erases the diffs. It does exactly:

- Read all three diffs (`git -C "$BENCH_CELLS_DIR"/conduit-<size>-<letter> diff bench-baseline`), labeled A, B and C.
- Read the requirements and the conventions.
- Produce a **comparative** review:
  - **Ranking** best to worst, one line each.
  - **Head-to-head differences** — structure, layering, naming, edge cases, test coverage.
  - **Suspected bugs the deterministic harness cannot catch**, per solution (or "none found").
  - **One-line verdict per solution.**
- Write it to `$SANDBOXES_REPO/bench/reviews/<size>.md` — committed there, accumulating: read the existing file first, then **prepend** a `## <YYYY-MM-DD> — <size>` section so prior runs are preserved.

### 5. Measure + report + reset

```bash
node "$SANDBOXES_REPO"/bench/run-all.mjs capture --sizes <size>
```

This runs the app's build and its full test suite, injects the acceptance oracle for the grading pass only, runs the convention and blast-radius checks, runs the capture eval and the doctor (skipped for the stock arm), folds in each rubric, computes the Overall composite, appends a row per cell to `stats.jsonl` and updates `history.jsonl` in place for a re-measurement, regenerates `REPORT.md`, writes per-run snapshots to `runs/`, then **resets the cells**. Pass `--no-reset` to inspect them first.

### 6. Report

Print the size's three-column table including the Overall rows and the version line, surface the reviewer's ranking and verdict with the arm names now attached, and confirm the cells were reset.

© alfredoperez · MIT · alfredoperez/speckit-companion

Similar slash commands

All slash commands for Agent setup: memory, context, orchestration →

From the same set83

The whole set →
speckit.companion.living-drift speckit.companion.living-drift finds capability files that changed after their living spec was last committed. Use it to spot project changes that have not made it back into the spec. CX Codex +1 ★ 110repo stars speckit.companion.resume speckit.companion.resume is a command for Claude Code that resumes spec-driven development after a pause. It carries recorded decisions forward and starts the next pipeline step. CX Codex +1 ★ 110repo stars speckit.git.commit speckit.git.commit is a Claude Code command that stages and commits changes after Spec Kit commands, according to project configuration. CX Codex +1 ★ 110repo stars speckit-git-commit The speckit-git-commit skill stages and commits project changes around Spec Kit commands. Use it when Git history should be updated from the project’s Spec Kit settings. CX Codex +1 ★ 110repo stars speckit.companion.living-validate speckit.companion.living-validate is a command for Claude Code that checks living specs and feature deltas without editing files. It flags missing scenarios, unlinked headings, and syntax issues. CX Codex +1 ★ 110repo stars publish The publish command guides a VS Code extension release, from QA checks and changelog updates to versioning and pushing a tagged commit. A separate file is available for Codex. CX Codex +1 ★ 110repo stars release-loop release-loop is a command for Claude Code and Codex that moves a batch of changes through fixes, QA, and release. It prepares a report and waits for approval before publishing. CX Codex +1 ★ 110repo stars bench-wave bench-wave runs an end-to-end benchmark round for a pipeline-diet wave across two Companion cells and logs the metrics into a markdown plan. Useful for measuring speed, token counts, and driver feedback. CX Codex +1 ★ 110repo stars

Project sponsors

Companies that keep this analytics open