metric-pack-designer — Claude skill: custom metrics for plugin-eval
A skill that helps you describe a local metric pack for plugin-eval: rubric categories, checks and metrics in a schema-compatible shape.
npx skills add openai/plugins --skill metric-pack-designer -a codexWhat metric-pack-designer does
The skill kicks in when you want to extend plugin-eval with your own rubric. It starts by clarifying the rubric categories and target kinds, then narrows things down to the smallest useful checks[] and metrics[] payload. What you get is a metric-pack manifest plus a script that prints JSON to stdout.
Run the pack with plugin-eval analyze <path> --metric-pack <manifest.json>. A pack may emit only checks[], metrics[] and an optional artifacts[]; overwriting the core score or summary is not on the table. Keep IDs stable across runs, otherwise comparisons stop meaning much.
The skill's own guidance is to prefer deterministic local signals over subjective text generation. The manifest format is documented in ../../references/metric-pack-manifest.md.
What it helps with
- define a local evaluation rubric for plugin-eval
- build the smallest useful set of checks and metrics for your criteria
- produce a metric-pack manifest and a script that prints JSON to stdout
- run the pack through plugin-eval analyze and compare runs
Example prompts for the agent
- help me design a metric pack for plugin-eval around our quality criteria
- create the manifest and a script that prints checks and metrics as JSON
- run my metric pack through plugin-eval analyze
How to install metric-pack-designer
1Skills CLI (Vercel)
npx skills add openai/plugins --skill metric-pack-designer -a codex2Manually: put the files in place
.agents/skills/metric-pack-designer/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add openai/plugins --skill metric-pack-designer -a claude-code2Manually: put the files in place
~/.claude/skills/metric-pack-designer/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/metric-pack-designer/
Manually: put the files in place
.cursor/skills/metric-pack-designer/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install openai/plugins metric-pack-designer2Manually: put the files in place
.github/skills/metric-pack-designer/SKILL.mdPlatform CLI
gemini skills install https://github.com/openai/plugins --path plugins/plugin-eval/skills/metric-pack-designerManually: put the files in place
.devin/skills/metric-pack-designer/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: Codex skills: how to install, where they live, how to call them
Questions about metric-pack-designer
How do I install metric-pack-designer in Claude?
Run npx skills add openai/plugins --skill metric-pack-designer, then npx skills add openai/plugins --skill metric-pack-designer -a codex. The skill file lands at .agents/skills/metric-pack-designer/SKILL.md.
What does metric-pack-designer do?
It helps you add a custom rubric to plugin-eval: clarify categories, assemble checks[] and metrics[], and create a metric-pack manifest plus a script that prints JSON.
Can a metric pack override the core plugin-eval score?
No. A pack emits only checks[], metrics[] and an optional artifacts[], and the core score and summary cannot be overwritten.
Source text
SKILL.md · MIT
--- name: metric-pack-designer description: Design custom metric packs for plugin-eval so teams can add local evaluation rubrics that emit schema-compatible checks and metrics. Use when the user wants their own evaluation criteria or visualizations. --- # Metric Pack Designer Use this skill when the user wants to extend `plugin-eval` with a local rubric. ## Workflow 1. Clarify the custom rubric categories and target kinds. 2. Define the smallest useful `checks[]` and `metrics[]` payload. 3. Create a metric-pack manifest plus a script that prints JSON to stdout. 4. Run the pack through `plugin-eval analyze <path> --metric-pack <manifest.json>`. ## Design Rules - Keep IDs stable across runs so comparisons stay meaningful. - Emit only `checks[]`, `metrics[]`, and optional `artifacts[]`. - Do not try to overwrite the core score or summary. - Prefer deterministic local signals over subjective text generation. ## Reference - `../../references/metric-pack-manifest.md`
© OpenAI Codex · MIT · openai/plugins
