run-benchmark — Claude Code skill: benchmark and install
run-benchmark — a Claude Code skill that runs Evolving Infra replay benchmarks, scores the run, and compares it with reference results. It activates for requests to reproduce paper results or benchmark a change.
Apache-2.0 ★ 173 updated 2026-11-16
npx skills add mit-han-lab/EvolvingInfra --skill run-benchmark -a claude-codeWhat run-benchmark does
The skill runs a replay benchmark for claude-code, putnam, or osworld, scores the run, and compares it with reference results. Its output includes run details and a table for each window, with requests per second, prefix cache hits, latency, and failed requests. It also reports switch times and the reproduction command.
A run needs a benchmark config, a model, and cluster settings: a Slurm account, partition, and runtime image. The examples/slurm/submit.py script submits the job, and results go to the run directory. The job occupies nine GPU nodes for about two hours, so the owner must approve submission first. Without Slurm, start the stack with the deploy-stack skill, then run the replay and scoring commands.
Compare runs on the same time window and P/D split. The skill rejects results that fail its validity checks; use diagnose-run to investigate a failed or slower run.
What it helps with
- reproduce a paper's benchmark results
- measure the performance of a code change
- compare runs on matching windows and P/D splits
- build a performance table for a pull request
Example prompts for the agent
- Reproduce the paper's
claude-codebenchmark and compare the run with the reference. - Benchmark my change using
examples/gb300/glm52.yaml. - Create a performance table for this pull request from the benchmark run.
How to install run-benchmark
1Skills CLI (Vercel)
npx skills add mit-han-lab/EvolvingInfra --skill run-benchmark -a claude-code2Manually: put the files in place
~/.claude/skills/run-benchmark/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/run-benchmark/
1Skills CLI (Vercel)
npx skills add mit-han-lab/EvolvingInfra --skill run-benchmark -a codex2Manually: put the files in place
.agents/skills/run-benchmark/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add mit-han-lab/EvolvingInfra --skill run-benchmark -a cursor2Manually: put the files in place
.cursor/skills/run-benchmark/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install mit-han-lab/EvolvingInfra skills/run-benchmark2Manually: put the files in place
.github/skills/run-benchmark/SKILL.md1Skills CLI (Vercel)
npx skills add mit-han-lab/EvolvingInfra --skill run-benchmark -a gemini-cli2Platform CLI
gemini skills install https://github.com/mit-han-lab/EvolvingInfra --path skills/run-benchmarkManually: put the files in place
.devin/skills/run-benchmark/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about run-benchmark
How do I install run-benchmark in Claude Code?
Run npx skills add mit-han-lab/EvolvingInfra --skill run-benchmark -a claude-code. The skill file is placed at ~/.claude/skills/run-benchmark/SKILL.md.
What does run-benchmark do?
It runs an Evolving Infra replay benchmark, checks whether the run is valid, and compares its metrics with the reference. The output contains a results table, switch times, and the reproduction command.
What does run-benchmark need to run?
It needs a benchmark config, model directory, cluster settings, and shared storage. Submitting the job requires a Slurm account, partition, and runtime image; the run uses nine GPU nodes for about two hours.
Source text
SKILL.md · Apache-2.0
---
name: run-benchmark
description: Run the paper's replay benchmark on Evolving Infra, score it and compare it with the reference results. Use when asked to reproduce the paper's numbers, benchmark a change, or produce the performance table of a pull request. Not for finding why a run failed or is slower than the reference; use `diagnose-run`.
---
# Run the replay benchmark
Run one of the paper's replay benchmarks on Evolving Infra, score it, and compare it with the
reference results.
## Scope
- **Reads**: `benchmarks/`, `examples/`: the workload's README and `AGENTS.md`, the configs and
`examples/slurm/submit.py`.
- **Writes**: the run directory under `--output-dir`; nothing in the repository.
- **Never changes**: `evolving_infra/`, `benchmarks/`: a benchmark measures the code as it is,
and a run with a replay setting other than the reference's does not compare with it.
- **Outside the repository**: submits a Slurm job that holds nine GPU nodes for about two
hours (20 minutes to start, 90 minutes of replay). Confirm with the owner before submitting.
## Inputs
- The workload: [`claude-code`](../../benchmarks/claude-code/README.md),
[`putnam`](../../benchmarks/putnam/README.md) or
[`osworld`](../../benchmarks/osworld/README.md).
- A config: [`examples/gb300/glm52.yaml`](../../examples/gb300/glm52.yaml) for the paper's
Claude Code experiment.
- The cluster: Slurm account, partition, the runtime image.
- What the result is compared with.
## Steps
### 1. Submit
Read the workload's README and `AGENTS.md` first: they give the paper's setup, its results and
the rules of that workload. `submit.py --benchmark <name>` runs a benchmark of the config as
one job: `claude-code` (the default) and `putnam` in `examples/gb300/glm52.yaml`, `osworld`
in `examples/gb300/kimi-k26.yaml`.
```bash
python examples/slurm/submit.py --config <config> --account <a> --partition <p> --image <image> \
--mounts <model directory>:/models --extra-args "<more sbatch options>" --output-dir <shared dir>/runs --dry-run
```
Read the dry run: the `#SBATCH` line, the mounts (the model must be at the config's `model`
path inside the container), and the replay command (`run_trie.py` with `stack`, `workload_path`, `concurrency`,
`duration` and `output_dir`; every other replay setting is the paper's by default).
Then run it without `--dry-run`. `--output-dir` is on storage every node sees. Site settings
can also stay in a copy of the config (`slurm:` section). Without Slurm: start the stack with
the [`deploy-stack`](../deploy-stack/SKILL.md) skill, then run the replay and scoring commands of the dry
run on the host that runs the router.
**Check**: the dry run shows the `#SBATCH` line, the model mounted at the config's `model`
path, and the `run_trie.py` command; the submission prints a job ID.
### 2. Watch
Under `<output dir>/<job id>/`: `serve.log` (ends with `ready: ... (9 workers, 6P3D)`), one
directory per host with `logs/<process>.log`, `bench.log` (the replay client), and at the end
`score.json`. The host directory that holds `READY` is the one that runs the router and the
planner ("the head" below). The job stops the stack when the replay ends, so `srun: ...
Killed` lines at the end of `serve.log` are the normal stop. If `serve.log` shows
`<name> exited` or the job ends early, use the [`diagnose-run`](../diagnose-run/SKILL.md) skill.
**Check**: `serve.log` shows `ready: ... (9 workers, 6P3D)` and `score.json` exists when the
job ends.
### 3. Read the result
```bash
cat <run>/score.json # the scored window [warmup, duration)
python skills/run-benchmark/scripts/switch_times.py <run>/<head>/planner <run>/bench
```
The run is valid when all of these hold:
- `failed_requests` is 0 (`cancelled_requests` are requests cut off when the replay ends);
- `switch_times.py` exits 0 (the planner did not fail);
- `replay_differs_from_preset` is `{}`. A run with another replay setting does not compare
with the reference unless that setting is what is being tested.
`cache_hit` is the prefix hit: cached prompt tokens over prompt tokens.
**Check**: all three conditions above hold. Otherwise the run is not valid: report why and do
not compare it.
### 4. Compare
The reference is in [`examples/gb300/README.md`](../../examples/gb300/README.md): the
whole-window result, the reference run's switch times, and its result on a window of each
P/D split.
1. Compare the whole scored window.
2. Compare one window per split. `switch_times.py` gives this run's switches on the replay
clock; start a window 120 s after a switch and end it at the next one:
```bash
python benchmarks/replay/score.py <run>/bench --skip-s <switch + 120> --window-s <length>
```
A whole-window gap with equal per-split results is a gap in when the planner switched.
3. Three runs of the reference configuration differ by about 4 % in requests per second, on
the whole window and on the 6P3D window. For a larger gap on a like-for-like window, use
"Slower than the reference" in [`diagnose-run`](../diagnose-run/SKILL.md) before concluding.
**Check**: both runs are compared on the same window and the same P/D split.
## Returns
One line with the setup (model, nodes, starting split, workload, concurrency), then a table
with a row per window for the run and the reference: window, split, requests per second,
prefix hit, time to first token (p50, p95), mean end-to-end latency, failed requests. Then
both runs' switch times and the reproduction command.
© mit-han-lab · Apache-2.0 · mit-han-lab/EvolvingInfra