plugin-eval · wshobson — Claude Code plugin: checks and installation
Lint plugin and skill structure without model calls, then optionally run experimental model-based scoring. Use it to inspect or compare skills.
/plugin marketplace add wshobson/agents
/plugin install plugin-eval@claude-code-workflowsWhat plugin-eval does
The static layer checks structure and flags anti-patterns without calling a model. It alone supplies plugin scores and badges, including those in the weekly CI report. The plugin-eval compare command compares two skills.
An experimental LLM judge makes four model calls and returns four holistic scores from 0 to 1. The experimental Monte Carlo layer sends 50 to 100 prompts and tracks replies, response length, errors, and token use. Neither experimental layer has been validated against human labels; both require uv sync --extra llm. Static checks need no model calls.
Use score to evaluate a plugin or skill, or certify for a deep score and badge. In Claude Code, the available commands are /eval, /certify, and /compare. To prepare the CLI, run cd plugins/plugin-eval followed by uv sync; extended scoring will not run until you install the LLM extra.
What it helps with
- lint a plugin or skill structure
- flag anti-patterns without model calls
- compare two skills
- measure model replies across a prompt set
- assign a badge through deep scoring
Example prompts for the agent
- Run
/evalonpath/to/skillto check its structure. - Use
/compareto compare two skills. - Run
/certifyon a skill to score it at deep depth and assign a badge.
How to install plugin-eval
Plugin marketplace
/plugin marketplace add wshobson/agents/plugin install plugin-eval@claude-code-workflowsStep by step, with commands for each platform: Claude Code plugins: marketplaces, installing, and building your own
Questions about plugin-eval
How do I install plugin-eval in Claude Code?
Run /plugin marketplace add wshobson/agents, then /plugin install plugin-eval@claude-code-workflows. To prepare the CLI, enter plugins/plugin-eval and run uv sync.
What does plugin-eval do?
It checks plugin and skill structure, flags anti-patterns, and can compare two skills. Experimental modes add model-based scoring.
Does static linting call a model?
No. Static analysis runs without model calls. The experimental LLM judge and Monte Carlo layers require uv sync --extra llm.
Source text
plugin-eval · MIT
# plugin-eval plugin-eval scores Claude Code plugins and skills in up to three layers. The static layer is a lint. It's fast and deterministic, and it's useful for checking structure. The LLM judge and Monte Carlo layers are experimental, not validated against human labels. For the trace-based eval program, see `evals/README.md` at the repository root. ## Quick start ```bash cd plugins/plugin-eval uv sync # Static lint of a skill, with no model calls uv run plugin-eval score path/to/skill --depth quick # Static and LLM judge layers, experimental, needs `uv sync --extra llm` uv run plugin-eval score path/to/skill --depth standard # All three layers at deep depth (experimental) uv run plugin-eval certify path/to/skill ``` ## Layers 1. Static analysis runs structural checks and flags anti-patterns without calling a model. Plugin-level scores and badges, including the ones in the weekly CI report, come from this layer alone. 2. The LLM judge (experimental) makes four model calls and returns four holistic scores from 0 to 1. 3. Monte Carlo (experimental) sends 50 to 100 prompts to the model and records how often it replies, how long the replies are, how often it errors, and how many tokens it uses. ## Commands | CLI | Claude Code | Description | | --------------------- | ----------- | -------------------------------------------------- | | `plugin-eval score` | `/eval` | Score a plugin or skill | | `plugin-eval certify` | `/certify` | Score at deep depth and assign a badge | | `plugin-eval compare` | `/compare` | Compare two skills | | `plugin-eval init` | none | Write a corpus index, which no other command reads | ## Documentation See [docs/plugin-eval.md](../../docs/plugin-eval.md) for the full reference, which covers the layers, dimensions, scoring formula, anti-patterns, and project structure.
© Seth Hobson · MIT · wshobson/agents
Similar plugins
All plugins for Testing and QA →plugin-eval by other authors comes first.
