build-agent-evals — Claude Code skill for agent evals
The build-agent-evals Claude Code skill builds eval suites to test agent behavior. It turns failed sessions or written cases into tests and compares two runs.
MIT ★ 149 updated 2026-11-16
npx skills add jackulau/coding.sh --skill build-agent-evals -a claude-codeWhat build-agent-evals does
The skill uses eval tools from the coding.sh MCP server. It can find failed sessions suited to testing, create skill-trigger checks from prompts that should or should not activate a skill, or build tests from cases. A single session can also become its own task. The coding.sh MCP server is required.
Before writing a suite, the skill asks for approval. Files go either under evals/ in the repository or into the local coding.sh store. Session-based tests can include a verifier command such as npm test. The skill also compares experiments, reports the pass-rate difference, and identifies improved and regressed cases. It provides the test run command but does not run it.
What it helps with
- turn a failed agent session into a test
- check whether a skill triggers on the right prompts
- build an eval suite from written cases
- compare the outcomes of two agent runs
Example prompts for the agent
- Build eval tests from failed sessions in this repository.
- Check which prompts should trigger the
my-skillskill. - Compare two experiments and list improved and regressed cases.
How to install build-agent-evals
1Skills CLI (Vercel)
npx skills add jackulau/coding.sh --skill build-agent-evals -a claude-code2Manually: put the files in place
~/.claude/skills/build-agent-evals/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/build-agent-evals/
1Skills CLI (Vercel)
npx skills add jackulau/coding.sh --skill build-agent-evals -a codex2Manually: put the files in place
.agents/skills/build-agent-evals/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add jackulau/coding.sh --skill build-agent-evals -a cursor2Manually: put the files in place
.cursor/skills/build-agent-evals/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install jackulau/coding.sh packages/coding.sh/skills/build-agent-evals2Manually: put the files in place
.github/skills/build-agent-evals/SKILL.md1Skills CLI (Vercel)
npx skills add jackulau/coding.sh --skill build-agent-evals -a gemini-cli2Platform CLI
gemini skills install https://github.com/jackulau/coding.sh --path packages/coding.sh/skills/build-agent-evalsManually: put the files in place
.devin/skills/build-agent-evals/SKILL.mdThe legacy .windsurf/skills/ path still works
Claude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about build-agent-evals
How do I install build-agent-evals in Claude Code?
Run npx skills add jackulau/coding.sh --skill build-agent-evals -a claude-code. The skill file is at ~/.claude/skills/build-agent-evals/SKILL.md.
What does build-agent-evals do?
It creates eval suites from failed sessions, skill-trigger prompts, or written cases. It can also compare two experiments and show how their results differ.
What does build-agent-evals need to run?
It requires the coding.sh MCP server. The skill asks for approval before creating a suite, and session-based tests can include a verifier command.
Source text
SKILL.md · MIT
--- name: build-agent-evals description: "Builds eval suites with coding.sh from failed agent sessions, a skill's trigger prompts or cases you write, and reads paired comparisons of two runs. Use when the user wants an eval or regression test for their agent, wants to turn a failed session into a test, wants to check that a skill loads when it should, or asks which of two prompts, harnesses or models did better." license: MIT --- # Build evals for an agent Uses the coding.sh MCP server's eval tools. `eval_init` writes files under `evals/` in the repo and `eval_from_session` writes to the local coding.sh store. Ask the user before either, then pass `confirm: true`. ## Pick the source | The user wants | Call | |---|---| | Tests from runs that went wrong | `eval_candidates`, then `eval_init` (`source: "sessions"`) | | One session as a task | `eval_from_session` | | To check that a skill loads when it should | `eval_init` (`source: "skill"`) | | Cases for an app or a prompt | `eval_init` (`source: "cases"`) | Every `eval_init` call needs `name` (the suite's folder under `evals/`), `source` and `confirm: true`. ## From failed sessions 1. Call `eval_candidates`. It ranks this repo's recent sessions by how good a task each would make and rejects the ones that cannot be replayed. 2. Show the user the top candidates and the verifier you propose. A verifier is a shell command that exits 0 only when the task is solved, such as `npm test`. 3. After the user agrees, call `eval_init` with `name`, `source: "sessions"`, `sessionIds`, `verifier` and `confirm: true`. ## For a skill 1. Write prompts that should load the skill, 10 or more, and near misses that should not: requests that sound close but belong to another skill or to none. 2. After the user agrees, call `eval_init` with `name`, `source: "skill"`, `skill` (the skill's directory), `should`, `shouldNot` and `confirm: true`. ## From cases 1. Write the cases, each with an `input`. Give `expected` when a case has one exact answer. Otherwise write a `description`: it says what a good answer does and becomes the judge's rubric. 2. Find what answers a case: a `command` (it reads the input as JSON on stdin) or a `url`. 3. After the user agrees, call `eval_init` with `name`, `source: "cases"`, `cases`, the `command` or `url`, the `description` if you wrote one, and `confirm: true`. ## One session as a task After the user agrees, call `eval_from_session` with `sessionId` and `confirm: true`. Pass `verifier` when you know the command that proves the task is solved. ## After it is written - Give the user the `run` command from the result. Do not run it yourself: it starts agent runs that take minutes and can cost money. - When the result has `todo: true` or `notes`, tell the user what is left to fill in. - When the result has `ci`, offer it: it is the workflow that runs the suite on pull requests. ## Compare two runs 1. Call `experiments_list` to find the two experiments. 2. Call `experiment_compare` with `baseline` and `candidate`. 3. Give the verdict with the pass-rate difference and its interval, then the regressed and improved cases. When it reports a power warning, say the result needs more cases or trials. 4. Do not call a difference real when its interval includes zero.
© jackulau · MIT · jackulau/coding.sh