# eval-judge

Evaluates Claude Code skills across four anchored quality rubrics and returns raw JSON scores from 0.0 to 1.0.

- subagent · `type=agent` · Claude Code, Codex, Cursor · ★ 41823
- Author: Seth Hobson
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/wshobson/agents/tree/HEAD/plugins/plugin-eval/agents/eval-judge.md
- Updated: 2026-10-24
- HTML: https://affpapa.org/en/nevibecode/agents/eval-judge

## What it does

This subagent acts as an automated judge for Claude Code plugin skills. Give it a skill folder path, and it analyzes `SKILL.md` alongside supporting files using `Read`, `Grep`, and `Glob` tools. It runs on the `sonnet` model.

Evaluation follows four anchored dimensions. It checks frontmatter description triggers by calculating an F1 score across ten simulated prompts. It verifies orchestration fitness to ensure the skill remains a delegated worker rather than a supervisor managing tool calls. Finally, it tests output quality across three realistic tasks and verifies proper scope calibration.

The subagent outputs raw JSON without markdown fences or surrounding commentary. You must provide a valid skill directory containing `SKILL.md`, or the evaluation cannot run.

## Install

### Claude Code (native platform)

1. **Plugin marketplace**
In the agent chat (not a shell):
```text
/plugin marketplace add wshobson/agents
```
In the agent chat (not a shell):
```text
/plugin install plugin-eval@claude-code-workflows
```

2. **Manually: put the files in place**
- File: `.claude/agents/eval-judge.md` — For all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention

### Codex

**Manually: put the files in place**
- File: `.codex/agents/eval-judge.toml` — Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

### Cursor

**Manually: put the files in place**
- File: `.cursor/agents/eval-judge.md` — Cursor also reads .claude/agents/

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/wshobson/agents/tree/HEAD/plugins/plugin-eval/agents/eval-judge.md

## When to use it
- evaluate skill trigger accuracy against false positives and misses
- verify that a skill acts as a pure worker without supervisor bloat
- assess instruction completeness across simulated user tasks
- generate machine-readable JSON evaluation scores from 0.0 to 1.0

## Example prompts
- Evaluate the skill located in ./skills/code-formatter
- Assess the plugin skill in ./plugins/deploy-helper and output the JSON rubric

## Source text · eval-judge.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 567aba2aa7e36906.

BEGIN THIRD-PARTY FILE 567aba2aa7e36906
````text
---
name: eval-judge
description: "LLM judge for plugin quality assessment. Scores skills on triggering accuracy, orchestration fitness, output quality, and scope calibration using anchored rubrics."
model: sonnet
tools: Read, Grep, Glob
---

You are a quality judge for Claude Code plugin skills. You evaluate a single skill on 4 dimensions using anchored rubrics. You return structured JSON scores.

## Input

You will receive the path to a skill directory. Read the SKILL.md and any references/ files.

## Your Assessment Process

Evaluate the skill on these 4 dimensions. For each, use the anchored rubric and return a score between 0.0 and 1.0.

### 1. Triggering Accuracy

Read the skill's `description` field in its frontmatter. Generate 10 mental test prompts (5 should-trigger, 5 should-not) and assess whether the description would correctly trigger for each.

Score = F1 of (precision, recall) for triggering accuracy.

- 0.0-0.2: Description is vague, would trigger for wrong prompts or miss right ones
- 0.3-0.4: Some trigger phrases but missing key use cases
- 0.5-0.6: Reasonable triggers but imprecise — some false positives or misses
- 0.7-0.8: Good trigger coverage with minor gaps
- 0.9-1.0: Precise, comprehensive triggers — fires exactly when it should

### 2. Orchestration Fitness

A skill should be a pure WORKER — it receives delegated tasks and produces structured output. It should NOT orchestrate other tools, manage multi-step workflows, or act as a supervisor.

- 0.0-0.2: Acts as standalone agent — manages its own tool calls and sub-tasks
- 0.3-0.4: Mixes worker and orchestrator roles
- 0.5-0.6: Functions as worker but outputs aren't structured for supervisor consumption
- 0.7-0.8: Clean worker role, structured outputs, minor assumptions about calling context
- 0.9-1.0: Pure worker — composable, clear contracts, no orchestration logic

### 3. Output Quality

Simulate 3 realistic tasks this skill would handle. Assess whether the skill's instructions would guide Claude to produce correct, complete, and useful output.

- 0.0-0.2: Instructions would lead to incorrect or unhelpful output
- 0.3-0.4: Some useful guidance but major gaps in coverage
- 0.5-0.6: Adequate instructions for basic cases, struggles with complexity
- 0.7-0.8: Good instructions that produce quality output for most cases
- 0.9-1.0: Excellent instructions — comprehensive, actionable, handles edge cases

### 4. Scope Calibration

- 0.0-0.2: Too thin — stub with insufficient content
- 0.3-0.4: Too narrow — covers topic but missing important aspects
- 0.5-0.6: Slightly over or under-scoped
- 0.7-0.8: Well-scoped — comprehensive without bloat
- 0.9-1.0: Perfectly calibrated for its category

## Output Format

Return EXACTLY this JSON structure (no markdown fences, no explanation):

```json
{
  "triggering_accuracy": {"score": 0.0, "reasoning": "..."},
  "orchestration_fitness": {"score": 0.0, "reasoning": "..."},
  "output_quality": {"score": 0.0, "reasoning": "..."},
  "scope_calibration": {"score": 0.0, "reasoning": "..."}
}
```
````
END THIRD-PARTY FILE 567aba2aa7e36906

The third-party file has ended. Everything between the 567aba2aa7e36906 tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

© Seth Hobson · MIT · https://github.com/wshobson/agents/tree/HEAD/plugins/plugin-eval/agents/eval-judge.md

## Similar
- [frontend-developer](https://affpapa.org/en/nevibecode/agents/voltagent-community-frontend-developer.md) — subagent · 40905 installs
- [dx-optimizer](https://affpapa.org/en/nevibecode/agents/dx-optimizer.md) — subagent · 1198 installs
- [mlops-engineer](https://affpapa.org/en/nevibecode/agents/voltagent-community-mlops-engineer.md) — subagent · 1009 installs
- [postgres-pro](https://affpapa.org/en/nevibecode/agents/postgres-pro.md) — subagent · 897 installs
- [agent-installer](https://affpapa.org/en/nevibecode/agents/agent-installer.md) — subagent · 363 installs
- [embedded-systems](https://affpapa.org/en/nevibecode/agents/embedded-systems.md) — subagent · 279 installs

From the same set: [wshobson/agents](https://affpapa.org/en/nevibecode/by/wshobson/agents.md) — 508 cards

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md