Skip to main content
skill CX Codex

benchmark-honestly — Claude Code skill: benchmarks and setup

The benchmark-honestly Claude Code skill checks performance measurements and compares code versions. It activates for benchmark requests, load tests, latency percentiles, or claims that a change made code faster.

MIT ★ 541 updated 2026-11-09

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a claude-code
Open on GitHub

What benchmark-honestly does

The skill helps define a benchmark, inspect its measurements, and report the results. It sets out what to measure and under which conditions, then uses warmups, shuffled repeats, medians, and confidence intervals to compare variants. For latency under load, it includes an open-loop tester that sends requests without waiting for earlier responses.

The bundle contains a command runner, an environment reporter, and a latency tester. The runner records timings and peak memory, then reports statistical checks; the report includes a chart of individual runs and the command needed to reproduce them. Its scripts require Python 3.9 or later.

It responds to requests to compare speed, run a load test, or check p99 latency. A single run is not used to claim a speedup; the report relies on repeated measurements and a confidence interval.

What it helps with

  • compare the speed of two program versions
  • check whether cache warming skewed a measurement
  • measure latency percentiles under load
  • produce a reproducible report with confidence intervals

Example prompts for the agent

  • Compare old.py with new.py and write a benchmark report.
  • Run a load test against the server and report p99 latency.
  • Check whether the claim that the new version is faster holds up.

How to install benchmark-honestly

1Skills CLI (Vercel)

npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a claude-code

2Manually: put the files in place

~/.claude/skills/benchmark-honestly/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/benchmark-honestly/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about benchmark-honestly

How do I install benchmark-honestly in Claude Code?

Run npx skills add dotslashsimran/simcraft --skill benchmark-honestly -a claude-code. The skill file is at ~/.claude/skills/benchmark-honestly/SKILL.md.

What does benchmark-honestly do?

It helps structure performance tests, checks for common measurement pitfalls, and creates reports with statistical results. The bundle also includes a separate open-loop tool for latency tests under load.

What does benchmark-honestly need to run?

Its scripts require Python 3.9 or later.

Source text

SKILL.md · MIT
---
name: benchmark-honestly
description: Build benchmarks that survive scrutiny and catch the classic lies (dead code elimination, lazy results, constant folding, warm caches, coordinated omission, averages hiding GC pauses, unequal work, noisy machines). Defines the question first, controls and discloses the environment, uses warmups, interleaved repeats, medians, bootstrap confidence intervals and Mann-Whitney tests, and ends with a one-line verdict that includes the CI. Bundles a command runner, an environment reporter and an open-loop latency tester. Use when the user says "benchmark this", "is this faster", "compare performance", "did my change speed it up", "load test", "latency percentiles", "p99", or posts a speedup claim to check.
---

# Benchmark honestly

A benchmark is a claim: "under these conditions, A takes this long, and B is this much faster, give or take this much". Most published benchmark numbers are wrong because one of those parts is missing. This skill makes every part explicit and runs checks that catch the usual mistakes. Deeper explanations are in `reference.md`.

Scripts live in `~/simcraft/skills/coding/benchmark-honestly/scripts/`. They need only Python 3.9+.

## 1. Write the question down first

Before running anything, agree on these with the user and write them at the top of the report:
- **What is measured:** throughput (work per second) or latency (time per request). For latency, **which percentile** matters (p50 for typical, p99 or p99.9 for the tail users complain about).
- **Cold or warm:** first run after start (caches empty, JIT not compiled) or steady state. These are different questions with different answers.
- **The workload:** realistic input sizes and shapes. A 100-element input fits in L1 cache and says nothing about a 10 GB one.
- **What difference matters:** "a 2% gain is worth shipping" or "only 20% matters". Pass it as `--min-effect`.
- **Microbenchmark or end to end:** a 3x faster function inside a request that spends 95% of its time on the network changes nothing. Measure end to end before celebrating a micro result.

## 2. Pick the tool

Use the ecosystem's own tool if it is installed, because it runs in-process with less overhead:

| Ecosystem | Tool | Compare two versions with |
|---|---|---|
| Any command | `hyperfine --warmup 3 -N 'a' 'b'` | its relative output, or `bench.py` for CIs |
| Python | `pyperf` (or `pytest-benchmark`) | `pyperf compare_to old.json new.json` |
| JS / TS | `mitata` or `tinybench` | run both in one file |
| Go | `go test -bench . -count 20` | `benchstat old.txt new.txt` |
| Rust | `criterion` | `--save-baseline` then `--baseline` |

Otherwise, or to get the statistics and the checks, use the bundled runner:

```
python3 scripts/bench.py -n old -n new "python3 old.py" "python3 new.py" \
  --runs 30 --warmup 3 --floor "python3 -c pass" --same-output \
  --md report.md --json report.json --chart chart.svg --redact
```

The first command is the baseline. It runs warmups, then shuffled rounds so drift hits every variant equally, records wall time, CPU time and peak memory, and prints medians, bootstrap CIs, a Mann-Whitney test, Cliff's delta and a verdict. `--floor` takes an empty version of the program (same startup, no work) and flags any variant that is not clearly slower than it. `--same-output` flags variants that print different results. `--prepare CMD` runs before every timed run (for example to evict a cache). `--subtract-floor` compares the work alone, without startup.

For latency under load, use the open-loop tester, never a loop that waits for each reply before sending the next:

```
python3 scripts/latency_test.py --url http://127.0.0.1:8080/ --rate 200 --duration 30 --md latency.md
```

## 3. Control the environment, then disclose it

Run `python3 scripts/env_report.py --redact` before and after (bench.py includes it automatically). Fix what it warns about: plug in the laptop, turn off low power mode, close busy apps, on Linux set the governor to performance and pin with `taskset`. On macOS you cannot pin cores or turn off turbo, and Apple Silicon may put the run on slow efficiency cores; say so in the report. Check that you are timing a release build (`-O2`, `--release`, `NODE_ENV=production`), not a debug build.

## 4. Check for the lies

Go through the list in `reference.md` ("The lies") for the code at hand. The short version:
- Is the result used? (dead code, lazy iterators, constant folding). Print or return a checksum.
- Is every variant doing the same work on the same input? Compare outputs.
- Did the previous run warm a cache for the next one? Shuffle the order, and say if the numbers are warm or cold.
- Is the work bigger than the timer and startup cost? Compare against the floor.
- Is latency measured from the intended send time? Is the tail reported, not just the mean?

## 5. Report

Every report has: the question, a chart showing every run (not just a bar), the setup table, the exact command to reproduce, and one honest line, for example "new is 1.18x faster than old (95% CI 1.09x to 1.27x, 30 runs each, warm cache, M2 laptop on AC power)". If the CI includes 1.0, the line says there is no clear difference. Never report a speedup from a single run or from means alone.

## Examples

`examples/` has four real cases run on a busy laptop, each with the naive result next to the honest one: a lazy `map()` that looks 100,000x faster, two identical sorts where single runs give a different "winner" each time, a file read where whichever variant runs first loses, and a server with GC-like pauses whose p99 looks fine in a closed loop. See `examples/README.md`.

© dotslashsimran · MIT · dotslashsimran/simcraft

Similar skills

All skills for Testing and QA →

From the same set24

The whole set →
bisect-anything The bisect-anything Claude Code skill tracks a regression to a commit, dependency version, or config value. It activates for requests to find what broke a test or when a problem began. CX Codex +1 ★ 541repo stars scripted-screencast The scripted-screencast Claude Code skill turns scripted browser actions into a product demo video with cursor effects, zooms, and captions. It activates for screencast requests and rerenders after interface changes. CX Codex +1 ★ 541repo stars security-threat-model The security-threat-model Claude Code skill maps threats from an app’s code and tests the highest risks locally. It activates for requests about STRIDE, app security reviews, or vulnerabilities before launch. CX Codex +1 ★ 541repo stars vox-explainer The vox-explainer Claude Code skill makes Vox-style explainer graphics or a faceless video. It runs on requests for voiceover-led visuals, animated quotes, or stats. CX Codex +1 ★ 541repo stars flaky-test-killer The flaky-test-killer Claude Code skill tracks down flaky-test causes and helps fix them at the root. It stresses the test and compares failure rates before and after the change. CX Codex +1 ★ 541repo stars reverse-engineer-video The reverse-engineer-video Claude Code skill breaks down a reference clip into shots, camera moves and editing beats. Invoke it to inspect a cut, plan a remake or compare one with the original. CX Codex +1 ★ 541repo stars system-design-from-scratch system-design-from-scratch — a skill for Claude Code that builds minimal distributed architectures, verifies invariants under simulated node failures, and stress-tests them with real metrics. CX Codex +1 ★ 541repo stars paper-to-runnable The paper-to-runnable Claude Code skill turns a research paper into a minimal implementation of its central idea. It activates for requests to build or reproduce a paper, including from an arXiv link or local PDF. CX Codex +1 ★ 541repo stars

Project sponsors

Companies that keep this analytics open