# hugging-face-evaluation

hugging-face-evaluation adds evaluation results to Hugging Face model cards. Extracts benchmark tables from README, imports scores via Artificial Analysis API, runs custom evaluations with vLLM or lighteval.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode, Windsurf (Devin Desktop), Claude.ai · ★ 42059 repo stars
- Author: patchy631
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/patchy631/ai-engineering-hub/tree/HEAD/hugging-face-skills/skills/hugging-face-evaluation/SKILL.md
- Updated: 2026-10-12
- Security check: automated, 2026-11-16 (text) — no signs of malicious code found.
- HTML: https://affpapa.org/en/nevibecode/skills/hugging-face-evaluation

## What it does

The skill operates on the model-index metadata format. It parses markdown tables from model READMEs (selecting a specific table by number), pulls benchmarks through the Artificial Analysis API, and runs custom evaluations on GPU with a vLLM backend or via HF Jobs using inspect-ai. It generates model-index YAML entries, merges them into existing cards, and creates pull requests.

Required Python dependencies: `huggingface_hub`, `markdown-it-py`, `pyyaml`, `requests`. For local evaluation runs: GPU, `vllm`, `lighteval`, and `torch`. Importing from Artificial Analysis requires the `AA_API_KEY` environment variable. A Hugging Face token (`HF_TOKEN`) with write access is mandatory for creating PRs.

The `evaluation_manager.py` script checks for open PRs before creating a new one using the `get-prs` command. The `inspect-tables` command displays table structure in a README with numbers and columns. `extract-readme --table N` parses the chosen table and outputs YAML; flags `--apply` or `--create-pr` push changes.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add patchy631/ai-engineering-hub --skill hugging-face-evaluation -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/hugging-face-evaluation/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/hugging-face-evaluation/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add patchy631/ai-engineering-hub --skill hugging-face-evaluation -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/hugging-face-evaluation/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add patchy631/ai-engineering-hub --skill hugging-face-evaluation -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/hugging-face-evaluation/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install patchy631/ai-engineering-hub hugging-face-skills/skills/hugging-face-evaluation
```

2. **Manually: put the files in place**
- File: `.github/skills/hugging-face-evaluation/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add patchy631/ai-engineering-hub --skill hugging-face-evaluation -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/patchy631/ai-engineering-hub --path hugging-face-skills/skills/hugging-face-evaluation
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/hugging-face-evaluation/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add patchy631/ai-engineering-hub --skill hugging-face-evaluation -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/patchy631/ai-engineering-hub/tree/HEAD/hugging-face-skills/skills/hugging-face-evaluation/SKILL.md

## When to use it
- parse benchmark tables from model README and convert them to model-index YAML
- import model evaluation scores from Artificial Analysis API
- run model evaluation on GPU with vLLM or via HF Jobs
- merge new evaluation results into an existing model card without overwriting
- create a pull request with updated model-index in the model repository

## Example prompts
- Check for open PRs in `username/model-name` before creating a new one
- Show all tables in the README of `username/model` with their structure and columns
- Extract the second table from the README of `username/model` and generate model-index YAML

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 2decc4368d96b41a.

BEGIN THIRD-PARTY FILE 2decc4368d96b41a
````text
---
name: hugging-face-evaluation
description: Add and manage evaluation results in Hugging Face model cards. Supports extracting eval tables from README content, importing scores from Artificial Analysis API, and running custom model evaluations with vLLM/lighteval. Works with the model-index metadata format.
---

# Overview
This skill provides tools to add structured evaluation results to Hugging Face model cards. It supports multiple methods for adding evaluation data:
- Extracting existing evaluation tables from README content
- Importing benchmark scores from Artificial Analysis
- Running custom model evaluations with vLLM or accelerate backends (lighteval/inspect-ai)

## Integration with HF Ecosystem
- **Model Cards**: Updates model-index metadata for leaderboard integration
- **Artificial Analysis**: Direct API integration for benchmark imports
- **Papers with Code**: Compatible with their model-index specification
- **Jobs**: Run evaluations directly on Hugging Face Jobs with `uv` integration
- **vLLM**: Efficient GPU inference for custom model evaluation
- **lighteval**: HuggingFace's evaluation library with vLLM/accelerate backends
- **inspect-ai**: UK AI Safety Institute's evaluation framework

# Version
1.3.0

# Dependencies

## Core Dependencies
- huggingface_hub>=0.26.0
- markdown-it-py>=3.0.0
- python-dotenv>=1.2.1
- pyyaml>=6.0.3
- requests>=2.32.5
- re (built-in)

## Inference Provider Evaluation
- inspect-ai>=0.3.0
- inspect-evals
- openai

## vLLM Custom Model Evaluation (GPU required)
- lighteval[accelerate,vllm]>=0.6.0
- vllm>=0.4.0
- torch>=2.0.0
- transformers>=4.40.0
- accelerate>=0.30.0

Note: vLLM dependencies are installed automatically via PEP 723 script headers when using `uv run`.

# IMPORTANT: Using This Skill

## ⚠️ CRITICAL: Check for Existing PRs Before Creating New Ones

**Before creating ANY pull request with `--create-pr`, you MUST check for existing open PRs:**

```bash
uv run scripts/evaluation_manager.py get-prs --repo-id "username/model-name"
```

**If open PRs exist:**
1. **DO NOT create a new PR** - this creates duplicate work for maintainers
````
END THIRD-PARTY FILE 2decc4368d96b41a

The third-party file has ended. Everything between the 2decc4368d96b41a tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2125 of 22440 bytes. © patchy631 · MIT · https://github.com/patchy631/ai-engineering-hub/tree/HEAD/hugging-face-skills/skills/hugging-face-evaluation/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md
