Skip to main content
skill CX Codex

leaderboard · shpigford — Claude Code skill: what it does, how to install

Publishing step for the nurb leaderboard: audit the runs merged since the last regeneration, refresh the verdicts, regenerate the report and the public page, open the PRs.

MIT 124 installs ★ 653 updated 2026-11-02

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add Shpigford/nurb --skill leaderboard -a claude-code
Open on GitHub

What leaderboard does

Merging a submission PR changes nothing on the site. This skill is the step that publishes it: it lists run directories added to Shpigford/nurb-benchmarks after site/benchmarks.html was last regenerated and works through them.

The audit is concrete. Every row of results.jsonl needs harness, model, effort, seed, benchmark_revision and timeout_s. Each row also needs transcript.txt.gz and its part source, with no /Users/, /home/ or usernames left after decompression. Submitted parts get hashed against tests/solutions/; an exact match with a reference solution disqualifies the run. At least one row per run is rebuilt and re-graded, and the committed score has to come back identical, because grading is deterministic. One transcript per new contributor gets read as well.

Verdicts sit in VERDICTS inside src/nurb_evals/site.py. Regeneration takes uv run python -m nurb_evals.report --write and uv run python -m nurb_evals.site --out <this-repo>/site/benchmarks.html. Output goes out as two PRs, never straight to main. A clone of the benchmarks repo outside this project is required, and a stale venv needs uv sync --locked first.

What it helps with

  • audit every new benchmark run before it goes public
  • re-grade a suspicious row and compare the score
  • write verdicts for newly merged model and effort combos
  • regenerate REPORT.md and the public benchmark page
  • open publish PRs listing accepted and rejected runs

Example prompts for the agent

  • Update the leaderboard from the submission PRs I just merged.
  • Publish the benchmarks: check the runs added since the last regeneration.
  • Regenerate the benchmark page and open the PR.

How to install leaderboard

1Skills CLI (Vercel)

npx skills add Shpigford/nurb --skill leaderboard -a claude-code

2Manually: put the files in place

~/.claude/skills/leaderboard/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/leaderboard/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about leaderboard

How do I install leaderboard in Claude Code?

Run npx skills add Shpigford/nurb --skill leaderboard -a claude-code. The skill file lands at ~/.claude/skills/leaderboard/SKILL.md.

What does leaderboard do?

It takes the benchmark runs merged since the last publish, sanity-checks them, refreshes the editorial verdicts, and regenerates REPORT.md plus site/benchmarks.html as two PRs.

Does merging a submission PR update the page on its own?

No. Submission PRs are pure additions; the public page only changes after the skill runs and its publish PRs merge.

Source text

SKILL.md · MIT
---
name: leaderboard
description: Publish the nurb model leaderboard from merged benchmark submissions. Sanity-checks every run landed since the last regeneration, writes or refreshes the editorial verdicts, regenerates the benchmarks repo's REPORT.md and this repo's site/benchmarks.html, and opens the publish PRs. Use when the user says "update the leaderboard", "publish the benchmarks", "regenerate the benchmark page", or after merging submission PRs.
---

# nurb leaderboard

Two generated surfaces, one editorial layer, two repos. The benchmark (tasks, scorer, submissions, `REPORT.md`) lives in [Shpigford/nurb-benchmarks](https://github.com/Shpigford/nurb-benchmarks); the public page (`site/benchmarks.html`, behind nurb.app/benchmarks) lives here. Both surfaces regenerate mechanically from that repo's `submissions/`; the verdict sentences and subscription labels live in its `src/nurb_evals/site.py` and are written by a person. Submission PRs are pure additions and merge freely; nothing reaches the public page until this skill runs. That gap is deliberate: it is where the sanity check and the verdicts happen, so publishing is an editorial act, not a side effect of merging.

## Step 0: Get the benchmarks checkout

Clone or update `Shpigford/nurb-benchmarks` somewhere outside this repo (for example `../nurb-benchmarks`, or a temp directory), on its main. Everything below that touches submissions, verdicts, or `REPORT.md` happens in that checkout; only the final `site/benchmarks.html` lands here.

## Step 1: Find what is new

The runs added since the page was last regenerated. In this repo, `git log -1 --format=%ci -- site/benchmarks.html` dates the last publish; in the benchmarks checkout, list the submission directories whose merge commits landed after it (`git log --since=<date> --diff-filter=A --name-only -- submissions/ | grep results.jsonl`).

Each directory under `submissions/` is one run: `<harness>-<model>-<effort>-<hex>/` holding `results.jsonl` plus per-trial gzipped transcripts and part sources. If nothing is new, say so and stop.

## Step 2: Sanity-check every new run

Work from the benchmarks checkout (its own uv project; `uv sync --locked` first if the venv is stale). For each new run directory:

- **Rows parse and carry full identity**: every line of `results.jsonl` has harness, harness_version, model, effort, seed, nurb_version, benchmark_version, a 12-char benchmark_revision, and timeout_s. A benchmark_revision that matches no revision the benchmark ever shipped is disqualifying (revisions from before the August 2026 repo split shipped from Shpigford/nurb's history).
- **Artifacts are complete and sanitized**: a `transcript.txt.gz` and the part source for every row; no `/Users/`, `/home/`, or usernames anywhere once decompressed (the suite's sanitization test enforces this too).
- **The parts are authored, not planted**: hash the submitted part files against `tests/solutions/` and against parts from other submissions. An exact match with a reference solution is disqualifying; matches across unrelated submissions are worth reading.
- **Spot-check by re-grading**: for at least one row per new run (and every row that looks too good), rebuild the trial project (`task.materialize`, drop the part in, restore the submitted `measurements.toml` for leg_cup) and run the grader. The committed score must reproduce exactly; grading is deterministic.
- **Read one transcript** per new contributor (`gzcat <trial>/transcript.txt.gz`): the headless preamble held (no `nurb dev`), the model actually iterated, and the session matches the row's timings.

A run that fails a check is removed with a PR comment saying which check and why, not silently. Suspicion is not proof: when a re-grade mismatches, check the benchmark_revision first; a row graded under an older shipped revision reproduces under that revision's scorer, not today's.

## Step 3: Verdicts

Every (harness, model, effort) combo on the board should have an entry in `VERDICTS` in the benchmarks repo's `src/nurb_evals/site.py`: which subscription it runs on, and one or two sentences a person with a printer can act on. Ground every claim in the rows and transcripts (findings, timings, the honesty tasks); never speculate. A new combo without a verdict renders numbers-only, which is acceptable for a day, not a policy. Capped or censored data is named as such ("hit the session limit"), never averaged into a claim.

## Step 4: Regenerate, look, publish

From the benchmarks checkout:

```bash
uv run python -m nurb_evals.report --write
uv run python -m nurb_evals.site --out <this-repo>/site/benchmarks.html
```

Open `site/benchmarks.html` in a browser and look at it before publishing: label collisions on the chart, a card wrapping badly, an empty state showing when rows exist. Screenshot, not DOM-query. Then two PRs, never straight to main: in nurb-benchmarks, the regenerated `REPORT.md` plus any verdict edits and removed runs; here, the regenerated `site/benchmarks.html`. Each PR body lists which runs were published and which were rejected with reasons. `uv run pytest -q tests/test_report.py tests/test_pricing.py tests/test_contribute.py` in the benchmarks checkout must be green; the full grading suite only guards scorer code, which this skill never touches.

The page deploys with `site/` however the site deploys; this skill's job ends at the merged PRs.

© Shpigford · MIT · Shpigford/nurb

Similar skills

All skills for AI and ML →

leaderboard by other authors comes first.

leaderboard Karan Bansal leaderboard prints a per-repo ranking of the files costing the most context tokens, with fixes suggested for the worst offenders. CX Codex +1 ★ 586repo stars checkpoint-promotion Seth Hobson checkpoint-promotion gates a trained checkpoint before release by checking data quality, capability drift, forgetting, and paired results against the base model. CX Codex +1 2,600installs mandela LilMGenius mandela — a Claude Code skill that checks evaluations, metrics, and experiments for leakage. Run it before relying on an A/B test, holdout, score, or validation result. CX Codex +1 1,500installs cli-eval cheaper-inference cli-eval runs model evaluation suites directly from the CLI and generates scorecards. Helps catch LLM regressions before merging code. CX Codex +1 1,300installs tally-rewind phpstan tally-rewind is a Claude Code skill that revisits past PHP performance fixes and checks whether later releases lost their gains. It compares current code with benchmarks from the release history. CX Codex +1 ★ 82repo stars senior-prompt-engineer Alireza Rezvani senior-prompt-engineer — a skill for systematic prompt optimization via eval sets, RAG quality measurement, and agent validation. Works with any model, uses only stdlib Python. CX Codex +1 970installs logfire-evals pydantic logfire-evals is a Claude Code skill for running offline task evaluations in Python or Node.js and viewing reports in Logfire. It comes into play when setting up test datasets, agent checks, or LLM judges. CX Codex +1 734installs

From the same set4

Project sponsors

Companies that keep this analytics open