# bench-experiment

bench-experiment is a Claude Code command that runs a Conduit benchmark round and compares results with earlier sweeps to measure changes across selected cells.

- command · `type=command` · Claude Code, Codex, GitHub Copilot, Gemini CLI · ★ 110 repo stars
- Author: alfredoperez
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/alfredoperez/speckit-companion/tree/HEAD/.claude/commands/bench-experiment.md
- Updated: 2026-10-19
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/commands/bench-experiment

## What it does

The command prepares benchmark cells, dispatches drivers, captures metrics, and compares work, oracle, and wall-clock results with earlier runs. It then records the round in `speckit-sandboxes`, including `stats.jsonl`, `history.jsonl`, `REPORT.md`, run data, and reviews.

Before baking cells, it checks for the indexing marker and active Spotlight work. If the marker is missing or Spotlight is busy, the round waits while the user runs the specified `mdutil` command. Judges must wait until capture finishes, then run in small batches. Running them alongside capture has previously brought the machine down.

Install it by saving the command file at `.claude/commands/bench-experiment.md`. The experiment description can include options for sizes, modes, extension version, spec-kit version, driver model, and GUI preamble. When comparing the stock arm with Companion, use `work`: `overall` includes lifecycle capture, which the stock arm does not have.

## Install

### Claude Code (native platform)

**Manually: put the files in place**
- File: `.claude/commands/bench-experiment.md` — Call it with /bench-experiment. For all projects use ~/.claude/commands/

### Codex

**Manually: put the files in place**
- File: `~/.codex/prompts/bench-experiment.md` — Call it with /prompts:bench-experiment. Codex marks this format deprecated in favour of skills

### GitHub Copilot

**Manually: put the files in place**
- File: `.github/prompts/bench-experiment.prompt.md`

### Gemini CLI

**Manually: put the files in place**
- File: `.gemini/commands/bench-experiment.toml` — Gemini stores commands as TOML: the prompt text goes into the prompt field

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/alfredoperez/speckit-companion/tree/HEAD/.claude/commands/bench-experiment.md

## When to use it
- compare benchmark results with earlier sweeps
- test an unreleased extension build across selected cell sizes
- capture work, oracle, and wall-clock metrics
- save a benchmark round and its report in `speckit-sandboxes`

## Example prompts
- Run a benchmark experiment called “hook anchors” across all sizes and compare it with earlier sweeps.
- Repeat the baseline using `--ext code` and report the metrics.
- Compare the Companion and stock arms using the `work` metric.

## Source text · bench-experiment.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 552da325f11cd570.

BEGIN THIRD-PARTY FILE 552da325f11cd570
````text
---
allowed-tools: Bash(. .claude/sandboxes-env.sh:*), Bash(node "$SANDBOXES_REPO"/bench/sync-templates.mjs:*), Bash(node "$SANDBOXES_REPO"/bench/run-all.mjs:*), Bash(npm run compile:*), Bash(git -C:*), Bash(date:*), Bash(ps:*), Agent, Workflow, AskUserQuestion
description: Run one full benchmark round on Conduit and report it against previous sweeps
---

## Your task

Run one complete round of the benchmark and report what changed. The user says "run an experiment and bench it" and expects the whole thing without being asked for details.

The harness lives in the `bench/` folder of the sibling [`speckit-sandboxes`](https://github.com/alfredoperez/speckit-sandboxes) repo and the app it measures in [`conduit`](https://github.com/alfredoperez/conduit). `bench/README.md` there is the reference; this command is the procedure.

### Paths

Shell state does not persist between Bash calls, so start each one with this line:

```bash
. .claude/sandboxes-env.sh && eval "$(node "$SANDBOXES_REPO"/bench/paths.mjs --sh)"
```

It sets `$SANDBOXES_REPO` (the sibling `speckit-sandboxes` checkout, whose `bench/` folder is the harness) and `$BENCH_CELLS_DIR` (where the cells are baked). When you hand a path to an agent, expand it first: an agent gets the absolute path, never the variable.

### 1. Name the sweep

`$ARGUMENTS` is what the experiment is testing, plus optional flags:

| Flag | Default | What it does |
|---|---|---|
| `--sizes easy,hard` | all four | which sizes the round covers |
| `--modes speckit,companion` | all arms | which arms to run |
| `--ext code\|latest\|<tag>` | `code` | `code` = the working tree, `latest` = the published build, or a release tag to backtrack to |
| `--speckit latest\|keep` | `latest` | pin or refresh spec-kit itself |
| `--model sonnet\|opus` | the session's | model the driver agents run on |
| `--preamble on\|off` | `off` | whether drivers are handed the GUI preamble |

**`--preamble on` is a promise the drivers have to keep.** The bake records it, nothing enforces it: a round baked `on` whose drivers were handed nothing has a column that lies, which is worse than no column. When it is on, every driver prompt starts with the output of `node "$SANDBOXES_REPO"/bench/preamble.mjs creation <speckit|companion> <specDir>` — the same text the GUI sends — and the arm decides which workflow name it takes. When it is off, drivers dispatch the command bodies alone.

Pass `--sizes`/`--modes` straight through to `sync-templates.mjs`, `prep` and `capture` — the same list at every step, or the round measures cells it never ran. Pass `--model` as the `model` option on each driver Agent. Everything else in `$ARGUMENTS` is what the experiment is testing. Turn it into a short label — "hook anchors", "living specs off", "fresh baseline". If there is nothing to name, use today's date and call it a baseline repeat, which is also useful: it is how the noise floor gets measured.

### 2. Check the machine first

Twelve clones of a React app are about 8GB of files Spotlight has never seen, and indexing them has taken this machine to a load average of 260. Before baking:

```bash
ls "$BENCH_CELLS_DIR"/.metadata_never_index || echo "MISSING"
```

If it is missing, or if `ps aux | grep mds_stores` shows Spotlight busy, tell the user to run `sudo mdutil -i off <the cells directory>` (the expanded `$BENCH_CELLS_DIR`) and wait for them. Do not start a round on a machine that is already struggling.

### 3. Bake

```bash
npm run compile                          # the driver dispatches the GUI preamble from dist/
node "$SANDBOXES_REPO"/bench/sync-templates.mjs --sizes <sizes> --ext <ext> --speckit <speckit> --sweep "<label>"
```
````
END THIRD-PARTY FILE 552da325f11cd570

The third-party file has ended. Everything between the 552da325f11cd570 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 3692 of 6810 bytes. © alfredoperez · MIT · https://github.com/alfredoperez/speckit-companion/tree/HEAD/.claude/commands/bench-experiment.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
