Skip to main content
skill CX Codex

evaluate-skill · edonadei — Claude Code skill: evaluation and setup

The evaluate-skill Claude Code skill runs and interprets Caliper evaluations for skills. It checks activation, task success, and results against a run without the skill.

MIT ★ 237 updated 2026-11-16

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add edonadei/caliper --skill evaluate-skill -a claude-code
Open on GitHub

What evaluate-skill does

The skill runs Caliper evaluations, explains their results, and traces failures to the part that needs a fix. It can help write an .eval.yaml spec once the tasks have been chosen, or compare a full run with a control run that leaves out the skill being tested.

The evaluation keeps skill activation separate from task success. The caliper CLI must be available on PATH; if it is missing, the instructions give pipx install caliper-eval as the installation command. caliper validate <spec> checks a spec without network access.

Run evaluations with caliper run and compare results with caliper compare. Environment failures, unusable attempts, and activation problems are treated separately from a low task score.

What it helps with

  • check whether a skill activates for the intended prompts
  • measure task success with a Caliper spec
  • compare a skill run with a control that omits it
  • trace evaluation failures to the part that needs a fix

Example prompts for the agent

  • Run the evaluation in eval.eval.yaml and explain what the results show.
  • Compare the latest run with a control that omits the skill using caliper compare.
  • Write an .eval.yaml spec for the skill evaluation tasks I have chosen.

How to install evaluate-skill

1Skills CLI (Vercel)

npx skills add edonadei/caliper --skill evaluate-skill -a claude-code

2Manually: put the files in place

~/.claude/skills/evaluate-skill/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/evaluate-skill/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about evaluate-skill

How do I install evaluate-skill in Claude Code?

Run npx skills add edonadei/caliper --skill evaluate-skill -a claude-code. The skill file is at ~/.claude/skills/evaluate-skill/SKILL.md.

What does evaluate-skill do?

It runs and interprets Caliper evaluations for skills, covering activation, task success, and comparison with a control run that omits the skill.

What does evaluate-skill need to run?

The caliper CLI must be available on PATH. If it is missing, install it with pipx install caliper-eval.

Source text

SKILL.md · MIT
---
name: evaluate-skill
description: Run, read, and diagnose a skill's Caliper eval — its success rate over k attempts, whether it fires, and whether it beats the agent without it. Use when the user wants to run, validate, interpret, or compare a skill's eval, or write an .eval.yaml spec whose tasks they have already decided.
allowed-tools: Bash
---

# Evaluate Skill

Operate Caliper: run a skill's eval, read what the results say, and trace each failure to the place that fixes it.

## Prerequisites

The `caliper` CLI must be on `PATH`. This skill can be copied into an agent without the Caliper repo, so install it if missing:

```bash
pipx install caliper-eval
```

`caliper <command> --help` is the authority on flags. [REFERENCE.md](REFERENCE.md) covers the full spec format, engines, and what the results mean.

## The four questions

A skill eval answers four separate questions. Each has its own measurement and its own fix location, so keep them apart: a blended number hides which half broke.

| Question | Measured by | A failure is fixed in |
|---|---|---|
| **Fires**: does the agent reach for the skill when it should, and only then? | `activates:` on tasks, and trigger probes | the skill's `description` frontmatter |
| **Works**: once it fires, does it get the job done? | `expect:` / `assert:`, scored as the success rate | the skill's body |
| **Earns**: does it beat the agent without it? | the control (`--ablate <skill-name>`: the declared neighbourhood minus this skill) and `caliper compare`. For a truly bare agent, ablate every declared skill and every declared `mcp:` server, and isolate the run | the tasks: if the control passes too, the task is too easy |
| **Holds**: does it stay good across edits and over time? | `caliper compare` of each full run against the previous one, skill drift | the edit that moved it |

## Spec shape

```yaml
skills:
  - ./SKILL.md              # relative to the spec; installed, never preloaded
tasks:
  - name: What success looks like
    prompt: <what a real user would type; never names the skill>
    expect: <natural-language pass/fail criterion>
    assert: |               # optional deterministic Python check
      assert ...
    activates: [my-skill]   # frontmatter name: of the skill that should fire
  - name: Unrelated work stays unrelated
    prompt: <work no declared skill should answer>
    activates: []           # a trigger probe: no judge, cheap
```

When you write a spec, every task with `expect:` or `assert:` also asserts `activates:`, refusals included: the skill, plus any declared skill it delegates to on that task, and every prompt reads like a real user's request with the skill left unnamed.

The spec has no `backend`/`model` or `judge:` block. The engine is chosen at run time, independently for the skill and the judge: `caliper run <spec> --model codex` runs and grades on codex, and `--judge-model` picks a different judge. Backends are `claude-code` (default), `codex`, `pi`, `hermes`, `antigravity` and `opencode`. Each attempt runs in a fresh, empty workdir, so `setup:` builds fixtures there with relative paths.

## Running

1. `caliper validate <spec>`. It never touches the network.
2. `caliper run <spec> --k 1` to shake out spec and harness errors. Fix those before reading any score.
3. `caliper run <spec> --k 3 --ablate <skill-name>`, once (write `skill:<skill-name>` if an `mcp:` server shares the name). This is the **control**: the declared neighbourhood without this skill. The skill isn't installed, so editing `SKILL.md` can't move its number. Keep its results path (`caliper list <spec-name> --format json` gives each run's `path` and what it `ablated`) and re-diff against it. Re-run it only when the tasks or the declared skills change.
4. `caliper run <spec> --k 3`, then `caliper compare <control.json> <spec-name>` to see whether it earns its place. After each edit, also compare against the previous full run's path to see whether the edit held: a skill that got worse can still beat the control. A bare spec name resolves to that spec's latest run.

If a run stops with `Not logged in`, ask the user whether to run the login command the error names. With their consent, run it (it opens a browser for them to finish) and rerun; never run it without consent. `hermes model` is an interactive picker, so ask the user to run it in their own terminal instead.

## Reading results

The **success rate** (`successes / usable`) is the headline. pass@k and pass^k are secondary views under `--verbose`. Activation is a separate scoreboard, over a different population: report it beside the success rate, never averaged into it.

Trace every failing task to where its fix belongs:

| Signal | Where the fix belongs |
|---|---|
| Run exits `2` (backend misconfigured, unavailable model, failed hook or MCP server, or every attempt unusable) | The environment or the task's hooks. Report it as a configuration problem, not a task failure |
| `⊘` unusable attempts (`infra_error`, `timeout`, `judge_error`) | Not the skill: rate limits, auth, or the judge. They are excluded from the score; re-run |
| `cheat` outcome | The task: it leaks its answer. Tighten the task or `sandbox:` |
| Activation fails: an expected skill didn't fire | That skill's `description` |
| Activation fails: an unexpected skill fired too | The extra skill's `description`, or the overlap between the two |
| Activation fails because one of the user's own skills fired (`skill:` in the report header) | Not the `description` alone: that skill is real competition in their setup. Decide with the user whether to sharpen the description or isolate the run |
| Activation passes, score low | The skill's body |
| The judge's reasoning shows `expect:` was ambiguous | The task's grading: make the criterion observable, or add an `assert:` |
| Full run ≈ control, and the skill fired in the full run | The task: it doesn't need the skill. If the skill never fired, the fix is its `description` (above) |

At k=3 one attempt is a 33-point swing. Before calling a change a win or a regression, re-run at k≥5. `caliper compare` flags any drop and never gates. A rewritten or shortened skill is safe to ship when, at k≥5, it stays within about 5% of the previous score and still beats the control.

## Whose setup is measured

Runs load the user's own customizations by default (user skills, plugins, rules, settings and connectors; see REFERENCE.md for backend exceptions), which answers "does my skill work in *my* agent?". **Isolate** (`--no-user-customizations`, or `user_customizations: false` in the spec) when comparing backends or models, when the number leaves this machine (shared, published, compared with someone else's run), or when measuring the bare agent: each setup is different, so otherwise part of the delta is the setups. `--ablate` of the user's own skill needs no isolation, since both runs load the same setup.

**Always tell the user which mode ran** and what it loaded, from the report header's `user customizations:` line (absent means isolated), and relay any fix `caliper compare` suggests about it.

## No eval yet?

If the user hasn't decided what to test (a skill with no `.eval.yaml`, or an eval whose gaps they want found), suggest `grill-skill`: it interviews them and writes the spec. To design tasks yourself, follow "Designing evals" in [REFERENCE.md](REFERENCE.md).

## Done when

- every task has an observable criterion, and at least one has a deterministic `assert:`;
- execution tasks assert `activates:`, and the spec has at least one trigger probe;
- the full run beats the control on execution tasks, and holds against the previous full run. Trigger probes have no execution score: read them from the full run's activation;
- the spec passes `caliper validate`;
- the user has been told to commit the `.eval.yaml` beside `SKILL.md`. Saved runs under `.caliper/results/` are useful for diffing over time and safe to gitignore.

© edonadei · MIT · edonadei/caliper

Similar skills

All skills for Testing and QA →

evaluate-skill by other authors comes first.

evaluate-skill official OpenAI Codex evaluate-skill audits a local skill, explains the score, and points to the first fixes. Use it before benchmarking or measuring token usage. CX Codex GP ChatGPT +1 1installs bmad-tea-evaluate bmad-code-org The bmad-tea-evaluate Claude Code skill runs behavioral evaluations of an agent and helps address gaps. It activates when asked to evaluate an agent or provide evidence that it behaves as expected. CX Codex +1 866installs evaluate-presets mikeyobrien evaluate-presets — a skill for Claude Code runs evaluation scripts across Ralph built-in presets and audits hat routing. Runs trigger after edits to preset definitions, event flows, or core agent prompts. CX Codex +1 ★ 3,206repo stars yao-meta-skill Yao Team The yao-meta-skill Claude Code skill builds, revises, and evaluates agent skills. It handles skill migration, packaging, routing checks, and release tests. CX Codex +1 686installs analyzer syahiidkamil analyzer examines blind skill comparisons to explain the winner’s advantage and suggest changes to the losing skill. Run it after the comparator finishes. CX Codex ★ 406repo stars evaluate-rag ai-evals-course evaluate-rag guides separate evaluation of RAG retrieval and generation, metric selection, and document chunking tests. CX Codex +1 1,100installs opik-evaluate comet-ml The opik-evaluate Claude Code skill builds and runs evaluations of an app or prompt through Opik. It examines agent and RAG quality, analyzes errors, and returns an experiment with scores and a link. CX Codex +1 122installs

From the same set3

Project sponsors

Companies that keep this analytics open