evaluation-lm-evaluation-harness — Claude Code skill: setup and use

What evaluation-lm-evaluation-harness does

The skill covers evaluation workflows built around lm-evaluation-harness. Its examples use benchmarks such as MMLU, GSM8K, HellaSwag, TruthfulQA, and HumanEval to measure different aspects of model performance. Scores and configuration details can be written to JSON files for later inspection.

The documented workflows include loading HuggingFace models, evaluating several tasks in one run, and testing custom checkpoints. A shell script named eval_checkpoint.sh illustrates recurring checks during training, while a PyTorch Lightning callback shows one way to connect evaluation to a training loop. Other examples collect scores from several models into a comparison table and chart progress across checkpoints.

The command examples require the lm-eval package, installed with pip install lm-eval. Hardware settings such as --device cuda:0 need to match the environment running the tests. HumanEval involves code execution, and the guide cautions against using MMLU for frequent checkpoint checks because its full evaluation takes considerably longer than the suggested quick tests.