benchmark-honestly — Claude Code skill: benchmarks and setup
The benchmark-honestly Claude Code skill checks performance measurements and compares code versions. It activates for benchmark requests, load tests, latency percentiles, or claims that a change made code faster.
MIT ★ 541 updated 2026-11-09
npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a claude-codeWhat benchmark-honestly does
The skill helps define a benchmark, inspect its measurements, and report the results. It sets out what to measure and under which conditions, then uses warmups, shuffled repeats, medians, and confidence intervals to compare variants. For latency under load, it includes an open-loop tester that sends requests without waiting for earlier responses.
The bundle contains a command runner, an environment reporter, and a latency tester. The runner records timings and peak memory, then reports statistical checks; the report includes a chart of individual runs and the command needed to reproduce them. Its scripts require Python 3.9 or later.
It responds to requests to compare speed, run a load test, or check p99 latency. A single run is not used to claim a speedup; the report relies on repeated measurements and a confidence interval.
What it helps with
- compare the speed of two program versions
- check whether cache warming skewed a measurement
- measure latency percentiles under load
- produce a reproducible report with confidence intervals
Example prompts for the agent
- Compare
old.pywithnew.pyand write a benchmark report. - Run a load test against the server and report p99 latency.
- Check whether the claim that the new version is faster holds up.
How to install benchmark-honestly
1Skills CLI (Vercel)
npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a claude-code2Manually: put the files in place
~/.claude/skills/benchmark-honestly/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/benchmark-honestly/
1Skills CLI (Vercel)
npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a codex2Manually: put the files in place
.agents/skills/benchmark-honestly/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a cursor2Manually: put the files in place
.cursor/skills/benchmark-honestly/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install dotslashsimran/simcraft skills/coding/benchmark-honestly2Manually: put the files in place
.github/skills/benchmark-honestly/SKILL.md1Skills CLI (Vercel)
npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a gemini-cli2Platform CLI
gemini skills install https://github.com/dotslashsimran/simcraft --path skills/coding/benchmark-honestlyManually: put the files in place
.devin/skills/benchmark-honestly/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about benchmark-honestly
How do I install benchmark-honestly in Claude Code?
Run npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a claude-code. The skill file is at ~/.claude/skills/benchmark-honestly/SKILL.md.
What does benchmark-honestly do?
It helps structure performance tests, checks for common measurement pitfalls, and creates reports with statistical results. The bundle also includes a separate open-loop tool for latency tests under load.
What does benchmark-honestly need to run?
Its scripts require Python 3.9 or later.
Source text
SKILL.md · MIT
---
name: benchmark-honestly
description: Build benchmarks that survive scrutiny and catch the classic lies (dead code elimination, lazy results, constant folding, warm caches, coordinated omission, averages hiding GC pauses, unequal work, noisy machines). Defines the question first, controls and discloses the environment, uses warmups, interleaved repeats, medians, bootstrap confidence intervals and Mann-Whitney tests, and ends with a one-line verdict that includes the CI. Bundles a command runner, an environment reporter and an open-loop latency tester. Use when the user says "benchmark this", "is this faster", "compare performance", "did my change speed it up", "load test", "latency percentiles", "p99", or posts a speedup claim to check.
---
# Benchmark honestly
A benchmark is a claim: "under these conditions, A takes this long, and B is this much faster, give or take this much". Most published benchmark numbers are wrong because one of those parts is missing. This skill makes every part explicit and runs checks that catch the usual mistakes. Deeper explanations are in `reference.md`.
Scripts live in `~/simcraft/skills/coding/benchmark-honestly/scripts/`. They need only Python 3.9+.
## 1. Write the question down first
Before running anything, agree on these with the user and write them at the top of the report:
- **What is measured:** throughput (work per second) or latency (time per request). For latency, **which percentile** matters (p50 for typical, p99 or p99.9 for the tail users complain about).
- **Cold or warm:** first run after start (caches empty, JIT not compiled) or steady state. These are different questions with different answers.
- **The workload:** realistic input sizes and shapes. A 100-element input fits in L1 cache and says nothing about a 10 GB one.
- **What difference matters:** "a 2% gain is worth shipping" or "only 20% matters". Pass it as `--min-effect`.
- **Microbenchmark or end to end:** a 3x faster function inside a request that spends 95% of its time on the network changes nothing. Measure end to end before celebrating a micro result.
## 2. Pick the tool
Use the ecosystem's own tool if it is installed, because it runs in-process with less overhead:
| Ecosystem | Tool | Compare two versions with |
|---|---|---|
| Any command | `hyperfine --warmup 3 -N 'a' 'b'` | its relative output, or `bench.py` for CIs |
| Python | `pyperf` (or `pytest-benchmark`) | `pyperf compare_to old.json new.json` |
| JS / TS | `mitata` or `tinybench` | run both in one file |
| Go | `go test -bench . -count 20` | `benchstat old.txt new.txt` |
| Rust | `criterion` | `--save-baseline` then `--baseline` |
Otherwise, or to get the statistics and the checks, use the bundled runner:
```
python3 scripts/bench.py -n old -n new "python3 old.py" "python3 new.py" \
--runs 30 --warmup 3 --floor "python3 -c pass" --same-output \
--md report.md --json report.json --chart chart.svg --redact
```
The first command is the baseline. It runs warmups, then shuffled rounds so drift hits every variant equally, records wall time, CPU time and peak memory, and prints medians, bootstrap CIs, a Mann-Whitney test, Cliff's delta and a verdict. `--floor` takes an empty version of the program (same startup, no work) and flags any variant that is not clearly slower than it. `--same-output` flags variants that print different results. `--prepare CMD` runs before every timed run (for example to evict a cache). `--subtract-floor` compares the work alone, without startup.
For latency under load, use the open-loop tester, never a loop that waits for each reply before sending the next:
```
python3 scripts/latency_test.py --url http://127.0.0.1:8080/ --rate 200 --duration 30 --md latency.md
```
## 3. Control the environment, then disclose it
Run `python3 scripts/env_report.py --redact` before and after (bench.py includes it automatically). Fix what it warns about: plug in the laptop, turn off low power mode, close busy apps, on Linux set the governor to performance and pin with `taskset`. On macOS you cannot pin cores or turn off turbo, and Apple Silicon may put the run on slow efficiency cores; say so in the report. Check that you are timing a release build (`-O2`, `--release`, `NODE_ENV=production`), not a debug build.
## 4. Check for the lies
Go through the list in `reference.md` ("The lies") for the code at hand. The short version:
- Is the result used? (dead code, lazy iterators, constant folding). Print or return a checksum.
- Is every variant doing the same work on the same input? Compare outputs.
- Did the previous run warm a cache for the next one? Shuffle the order, and say if the numbers are warm or cold.
- Is the work bigger than the timer and startup cost? Compare against the floor.
- Is latency measured from the intended send time? Is the tail reported, not just the mean?
## 5. Report
Every report has: the question, a chart showing every run (not just a bar), the setup table, the exact command to reproduce, and one honest line, for example "new is 1.18x faster than old (95% CI 1.09x to 1.27x, 30 runs each, warm cache, M2 laptop on AC power)". If the CI includes 1.0, the line says there is no clear difference. Never report a speedup from a single run or from means alone.
## Examples
`examples/` has four real cases run on a busy laptop, each with the naive result next to the honest one: a lazy `map()` that looks 100,000x faster, two identical sorts where single runs give a different "winner" each time, a file read where whichever variant runs first loses, and a server with GC-like pauses whose p99 looks fine in a closed loop. See `examples/README.md`.
© dotslashsimran · MIT · dotslashsimran/simcraft