leaderboard · shpigford — Claude Code skill: what it does, how to install
Publishing step for the nurb leaderboard: audit the runs merged since the last regeneration, refresh the verdicts, regenerate the report and the public page, open the PRs.
MIT 124 installs ★ 653 updated 2026-11-02
npx skills add Shpigford/nurb --skill leaderboard -a claude-codeWhat leaderboard does
Merging a submission PR changes nothing on the site. This skill is the step that publishes it: it lists run directories added to Shpigford/nurb-benchmarks after site/benchmarks.html was last regenerated and works through them.
The audit is concrete. Every row of results.jsonl needs harness, model, effort, seed, benchmark_revision and timeout_s. Each row also needs transcript.txt.gz and its part source, with no /Users/, /home/ or usernames left after decompression. Submitted parts get hashed against tests/solutions/; an exact match with a reference solution disqualifies the run. At least one row per run is rebuilt and re-graded, and the committed score has to come back identical, because grading is deterministic. One transcript per new contributor gets read as well.
Verdicts sit in VERDICTS inside src/nurb_evals/site.py. Regeneration takes uv run python -m nurb_evals.report --write and uv run python -m nurb_evals.site --out <this-repo>/site/benchmarks.html. Output goes out as two PRs, never straight to main. A clone of the benchmarks repo outside this project is required, and a stale venv needs uv sync --locked first.
What it helps with
- audit every new benchmark run before it goes public
- re-grade a suspicious row and compare the score
- write verdicts for newly merged model and effort combos
- regenerate
REPORT.mdand the public benchmark page - open publish PRs listing accepted and rejected runs
Example prompts for the agent
- Update the leaderboard from the submission PRs I just merged.
- Publish the benchmarks: check the runs added since the last regeneration.
- Regenerate the benchmark page and open the PR.
How to install leaderboard
1Skills CLI (Vercel)
npx skills add Shpigford/nurb --skill leaderboard -a claude-code2Manually: put the files in place
~/.claude/skills/leaderboard/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/leaderboard/
1Skills CLI (Vercel)
npx skills add Shpigford/nurb --skill leaderboard -a codex2Manually: put the files in place
.agents/skills/leaderboard/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add Shpigford/nurb --skill leaderboard -a cursor2Manually: put the files in place
.cursor/skills/leaderboard/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install Shpigford/nurb .claude/skills/leaderboard2Manually: put the files in place
.github/skills/leaderboard/SKILL.md1Skills CLI (Vercel)
npx skills add Shpigford/nurb --skill leaderboard -a gemini-cli2Platform CLI
gemini skills install https://github.com/Shpigford/nurb --path .claude/skills/leaderboardManually: put the files in place
.devin/skills/leaderboard/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add Shpigford/nurb --skill leaderboard -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about leaderboard
How do I install leaderboard in Claude Code?
Run npx skills add Shpigford/nurb --skill leaderboard -a claude-code. The skill file lands at ~/.claude/skills/leaderboard/SKILL.md.
What does leaderboard do?
It takes the benchmark runs merged since the last publish, sanity-checks them, refreshes the editorial verdicts, and regenerates REPORT.md plus site/benchmarks.html as two PRs.
Does merging a submission PR update the page on its own?
No. Submission PRs are pure additions; the public page only changes after the skill runs and its publish PRs merge.
Source text
SKILL.md · MIT
---
name: leaderboard
description: Publish the nurb model leaderboard from merged benchmark submissions. Sanity-checks every run landed since the last regeneration, writes or refreshes the editorial verdicts, regenerates the benchmarks repo's REPORT.md and this repo's site/benchmarks.html, and opens the publish PRs. Use when the user says "update the leaderboard", "publish the benchmarks", "regenerate the benchmark page", or after merging submission PRs.
---
# nurb leaderboard
Two generated surfaces, one editorial layer, two repos. The benchmark (tasks, scorer, submissions, `REPORT.md`) lives in [Shpigford/nurb-benchmarks](https://github.com/Shpigford/nurb-benchmarks); the public page (`site/benchmarks.html`, behind nurb.app/benchmarks) lives here. Both surfaces regenerate mechanically from that repo's `submissions/`; the verdict sentences and subscription labels live in its `src/nurb_evals/site.py` and are written by a person. Submission PRs are pure additions and merge freely; nothing reaches the public page until this skill runs. That gap is deliberate: it is where the sanity check and the verdicts happen, so publishing is an editorial act, not a side effect of merging.
## Step 0: Get the benchmarks checkout
Clone or update `Shpigford/nurb-benchmarks` somewhere outside this repo (for example `../nurb-benchmarks`, or a temp directory), on its main. Everything below that touches submissions, verdicts, or `REPORT.md` happens in that checkout; only the final `site/benchmarks.html` lands here.
## Step 1: Find what is new
The runs added since the page was last regenerated. In this repo, `git log -1 --format=%ci -- site/benchmarks.html` dates the last publish; in the benchmarks checkout, list the submission directories whose merge commits landed after it (`git log --since=<date> --diff-filter=A --name-only -- submissions/ | grep results.jsonl`).
Each directory under `submissions/` is one run: `<harness>-<model>-<effort>-<hex>/` holding `results.jsonl` plus per-trial gzipped transcripts and part sources. If nothing is new, say so and stop.
## Step 2: Sanity-check every new run
Work from the benchmarks checkout (its own uv project; `uv sync --locked` first if the venv is stale). For each new run directory:
- **Rows parse and carry full identity**: every line of `results.jsonl` has harness, harness_version, model, effort, seed, nurb_version, benchmark_version, a 12-char benchmark_revision, and timeout_s. A benchmark_revision that matches no revision the benchmark ever shipped is disqualifying (revisions from before the August 2026 repo split shipped from Shpigford/nurb's history).
- **Artifacts are complete and sanitized**: a `transcript.txt.gz` and the part source for every row; no `/Users/`, `/home/`, or usernames anywhere once decompressed (the suite's sanitization test enforces this too).
- **The parts are authored, not planted**: hash the submitted part files against `tests/solutions/` and against parts from other submissions. An exact match with a reference solution is disqualifying; matches across unrelated submissions are worth reading.
- **Spot-check by re-grading**: for at least one row per new run (and every row that looks too good), rebuild the trial project (`task.materialize`, drop the part in, restore the submitted `measurements.toml` for leg_cup) and run the grader. The committed score must reproduce exactly; grading is deterministic.
- **Read one transcript** per new contributor (`gzcat <trial>/transcript.txt.gz`): the headless preamble held (no `nurb dev`), the model actually iterated, and the session matches the row's timings.
A run that fails a check is removed with a PR comment saying which check and why, not silently. Suspicion is not proof: when a re-grade mismatches, check the benchmark_revision first; a row graded under an older shipped revision reproduces under that revision's scorer, not today's.
## Step 3: Verdicts
Every (harness, model, effort) combo on the board should have an entry in `VERDICTS` in the benchmarks repo's `src/nurb_evals/site.py`: which subscription it runs on, and one or two sentences a person with a printer can act on. Ground every claim in the rows and transcripts (findings, timings, the honesty tasks); never speculate. A new combo without a verdict renders numbers-only, which is acceptable for a day, not a policy. Capped or censored data is named as such ("hit the session limit"), never averaged into a claim.
## Step 4: Regenerate, look, publish
From the benchmarks checkout:
```bash
uv run python -m nurb_evals.report --write
uv run python -m nurb_evals.site --out <this-repo>/site/benchmarks.html
```
Open `site/benchmarks.html` in a browser and look at it before publishing: label collisions on the chart, a card wrapping badly, an empty state showing when rows exist. Screenshot, not DOM-query. Then two PRs, never straight to main: in nurb-benchmarks, the regenerated `REPORT.md` plus any verdict edits and removed runs; here, the regenerated `site/benchmarks.html`. Each PR body lists which runs were published and which were rejected with reasons. `uv run pytest -q tests/test_report.py tests/test_pricing.py tests/test_contribute.py` in the benchmarks checkout must be green; the full grading suite only guards scorer code, which this skill never touches.
The page deploys with `site/` however the site deploys; this skill's job ends at the merged PRs.
© Shpigford · MIT · Shpigford/nurb
Similar skills
All skills for AI and ML →leaderboard by other authors comes first.