agent-regression-evals — Claude Code plugin: catch agent regressions
Compares agent runs before and after a change and decides whether the merge is safe. Useful when one trial can't separate a regression from noise.
/plugin marketplace add alirezarezvani/claude-skills
/plugin install agent-regression-evals@claude-code-skillsWhat agent-regression-evals does
Agents are non-deterministic. A single trial per case won't tell you whether a drop is real, and a correct final answer can hide a tool call that should never have fired. This plugin takes repeated-trial results and turns them into a merge decision with the statistics attached.
Three scripts, all on the Python standard library. trajectory_checker.py grades tool-call paths in strict, unordered, subset and superset modes, matches arguments, and blocks forbidden tools in every mode. flake_detector.py sorts cases into stable and flaky from repeated trials and reports pass@k and pass^k with Wilson intervals. regression_gate.py averages trials within a case, compares cases with a t-based CI and an exact one-sided McNemar test, and returns BLOCK on a significant drop, a critical-case regression, a dropped case, or a quarantine entry missing an owner and expiry. Also included: the cs-eval-gatekeeper agent, /cs-agent-evals and /cs-eval-gate commands, three references, and a CI job example.
You can run the scripts directly; each supports --help, --sample and --output json. One thing to plan for: too few trials per case exits with code 4, so the gate won't guess from a handful of runs. Quarantine expires here — an entry without an owner and a date walks back into the gate on its own. Path checks default to superset, because strict path matching kills valid alternatives.
What it helps with
- tell a real agent regression from random spread
- catch an extra or forbidden tool call hidden behind a correct answer
- find cases that flip between identical trials
- block a merge on a significant pass-rate drop
- stop someone deleting a failing case to get a green CI
Example prompts for the agent
- /cs-agent-evals baseline.jsonl candidate.jsonl cases.json
- /cs-eval-gate baseline.jsonl candidate.jsonl
- run python scripts/flake_detector.py --sample and list the flaky cases
How to install agent-regression-evals
Plugin marketplace
/plugin marketplace add alirezarezvani/claude-skills/plugin install agent-regression-evals@claude-code-skillsStep by step, with commands for each platform: Claude Code plugins: marketplaces, installing, and building your own
Questions about agent-regression-evals
How do I install agent-regression-evals in Claude?
Add the marketplace first: /plugin marketplace add alirezarezvani/claude-skills. Then install the plugin: /plugin install agent-regression-evals@claude-code-skills.
What does agent-regression-evals do?
It compares a baseline run against a candidate run and decides whether the candidate is measurably worse. Three scripts cover tool-call paths, case stability, and the significance of a drop.
How is it different from prompt-governance and self-eval?
prompt-governance versions and registers prompts, and self-eval grades one session's work. This plugin gates a specific change to an agent, prompt, model or tool.
Source text
agent-regression-evals · MIT
# agent-regression-evals > "The pass rate went from 82% to 79%" is not a finding until you know whether > three points is outside the noise. Agents are non-deterministic. One trial per case cannot tell a regression from a coin flip, and a correct final answer can hide a tool call that should never have happened. This plugin turns repeated-trial eval results into a merge decision with the statistics behind it. ## Install ```bash /plugin marketplace add alirezarezvani/claude-skills /plugin install agent-regression-evals ``` ## Use ```bash /cs-agent-evals baseline.jsonl candidate.jsonl cases.json # full pass /cs-eval-gate baseline.jsonl candidate.jsonl # just the gate ``` Or run the scripts directly. Each has `--help`, `--sample` and `--output json`: ```bash cd skills/agent-regression-evals python scripts/trajectory_checker.py --sample python scripts/flake_detector.py --sample python scripts/regression_gate.py --sample ``` ## The three scripts | Script | Question | Exit codes | |---|---|---| | `trajectory_checker.py` | Did the agent take an acceptable path? strict / unordered / subset / superset, argument matching, **forbidden tools block in every mode** | 0 · 2 deviations · 3 bad input · **4 forbidden tool** | | `flake_detector.py` | Which cases are stable and which flip between identical trials? pass@k, **pass^k**, Wilson intervals, quarantine stubs | 0 · 2 flaky · 3 · **4 too few trials** | | `regression_gate.py` | Is the candidate measurably worse? Per-case averaging, paired t-based CI, exact one-sided McNemar, critical cases, dropped cases, quarantine validity, cost | 0 PASS · 2 WARN · 3 · **4 BLOCK** | ## Design decisions - **pass^k, not just pass@k.** A user meets every trial, not the best one. tau-bench showed a model at 61% pass^1 falling below 25% at pass^8. - **Average within a case, compare across cases.** Treating 5 trials x 40 cases as 200 independent samples overstates confidence (Miller 2024). - **Missing cases block.** Deleting a failing case is the cheapest way to "fix" a regression. - **Quarantine expires.** Every entry needs an owner and a date; expired entries re-enter the gate by themselves. - **Outcome first, path where it matters.** Trajectory checks default to `superset`, following Anthropic's warning that strict path checks fail valid alternatives. See `skills/agent-regression-evals/references/` for every formula and its source. ## Distinct from - `prompt-governance` — versions and registers prompts; this gates a change. - `self-eval` — grades one session's work. - `agenthub` eval — ranks parallel agents inside one session. ## License MIT
© Alireza Rezvani · MIT · alirezarezvani/claude-skills
