# evaluation-lm-evaluation-harness

This skill guides Claude Code through language-model benchmarking with lm-evaluation-harness, from running standard tests to comparing model scores and monitoring training checkpoints.

- skill · `type=skill` · Claude Code · 92 installs
- Author: Orchestra Research
- License: MIT · policy=link-only · description only; the file stays with the author
- Source: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/evaluation-lm-evaluation-harness
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/skills/evaluation-lm-evaluation-harness

## What it does

The skill covers evaluation workflows built around lm-evaluation-harness. Its examples use benchmarks such as MMLU, GSM8K, HellaSwag, TruthfulQA, and HumanEval to measure different aspects of model performance. Scores and configuration details can be written to JSON files for later inspection.

The documented workflows include loading HuggingFace models, evaluating several tasks in one run, and testing custom checkpoints. A shell script named `eval_checkpoint.sh` illustrates recurring checks during training, while a PyTorch Lightning callback shows one way to connect evaluation to a training loop. Other examples collect scores from several models into a comparison table and chart progress across checkpoints.

The command examples require the `lm-eval` package, installed with `pip install lm-eval`. Hardware settings such as `--device cuda:0` need to match the environment running the tests. HumanEval involves code execution, and the guide cautions against using MMLU for frequent checkpoint checks because its full evaluation takes considerably longer than the suggested quick tests.

## Install

### Claude Code (native platform)

1. **claude-code-templates**
```bash
npx claude-code-templates@latest --skill ai-research/evaluation-lm-evaluation-harness
```

2. **Skills CLI (Vercel)**
```bash
npx skills add davila7/claude-code-templates --skill evaluation-lm-evaluation-harness -a claude-code
```

3. **Manually: put the files in place**
- File: `~/.claude/skills/evaluation-lm-evaluation-harness/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/evaluation-lm-evaluation-harness/

### Codex

**Manually: put the files in place**
- File: `.agents/skills/evaluation-lm-evaluation-harness/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

**Manually: put the files in place**
- File: `.cursor/skills/evaluation-lm-evaluation-harness/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install davila7/claude-code-templates evaluation-lm-evaluation-harness
```

2. **Manually: put the files in place**
- File: `.github/skills/evaluation-lm-evaluation-harness/SKILL.md`

### Gemini CLI

**Platform CLI**
```bash
gemini skills install https://github.com/davila7/claude-code-templates --path cli-tool/components/skills/ai-research/evaluation-lm-evaluation-harness
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/evaluation-lm-evaluation-harness/SKILL.md` — The legacy .windsurf/skills/ path still works

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/davila7/claude-code-templates/tree/HEAD/cli-tool/components/skills/ai-research/evaluation-lm-evaluation-harness

## When to use it
- Run MMLU and GSM8K evaluations on a language model
- Compare several models using the same benchmark tasks
- Export benchmark metrics and model settings to JSON
- Monitor score changes across training checkpoints
- Plot evaluation scores over training steps

## Example prompts
- Set up `lm-eval` for my HuggingFace model and run `mmlu,gsm8k,hellaswag`.
- Compare the models listed in `models.txt` on `mmlu,gsm8k,hellaswag,truthfulqa` and save separate JSON results.
- Create a checkpoint evaluation workflow with `eval_checkpoint.sh` and chart the scores from the saved results.

## Similar
- [skill-creator](https://affpapa.org/en/nevibecode/skills/skill-creator.md) — skill · 414180 installs
- [writing-for-agents](https://affpapa.org/en/nevibecode/skills/writing-for-agents.md) — skill · 466719 installs
- [media-use](https://affpapa.org/en/nevibecode/skills/media-use.md) — skill · 1100000 installs
- [mcp-builder](https://affpapa.org/en/nevibecode/skills/mcp-builder.md) — skill · 130949 installs
- [subagent-driven-development](https://affpapa.org/en/nevibecode/skills/subagent-driven-development.md) — skill · 237746 installs
- [orchestration](https://affpapa.org/en/nevibecode/skills/orchestration.md) — skill · 350300 installs

From the same set: [davila7/claude-code-templates](https://affpapa.org/en/nevibecode/by/davila7/claude-code-templates.md) — 804 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md