model-evaluator — Claude Code subagent for model evaluation and setup
This subagent helps compare models for a task, plan benchmarks, and investigate quality regressions after deployment. Use it when a choice needs measurable criteria.
npx claude-code-templates@latest --agent ai-specialists/model-evaluatorWhat model-evaluator does
model-evaluator helps assess models against task-specific measures such as output quality, latency, and cost. It can shape a benchmark, select metrics, and review results. For a post-deployment check, it compares current performance with a stored baseline.
Before recommending or testing a model, the agent uses WebSearch to verify current model names and IDs. WebFetch is available for detailed pricing and leaderboard pages. It first asks for success criteria, budget, latency and throughput targets, data constraints, and candidate models. A recommendation without those requirements would be premature.
In Claude Code, the agent can use Read, Write, Edit, Bash, Glob, Grep, WebSearch, and WebFetch. To install with the listed method, run npx claude-code-templates@latest --agent ai-specialists/model-evaluator; the agent file is .claude/agents/model-evaluator.md.
What it helps with
- compare candidate models against task requirements
- design a benchmark and choose evaluation metrics
- check whether a model update caused quality regressions
- weigh model quality against inference cost
Example prompts for the agent
- Compare models for support-ticket summaries using my budget and latency targets.
- Plan a benchmark for code-generation models and choose suitable evaluation metrics.
- Compare current results with the baseline and find which inputs regressed.
How to install model-evaluator
1claude-code-templates
npx claude-code-templates@latest --agent ai-specialists/model-evaluator2Manually: put the files in place
.claude/agents/model-evaluator.mdFor all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention
Manually: put the files in place
.codex/agents/model-evaluator.tomlCodex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/
Manually: put the files in place
.cursor/agents/model-evaluator.mdCursor also reads .claude/agents/
Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean
Questions about model-evaluator
How do I install model-evaluator in Claude Code?
Run npx claude-code-templates@latest --agent ai-specialists/model-evaluator. The agent file is .claude/agents/model-evaluator.md.
What does model-evaluator do?
It assists with model selection, benchmark design, and post-deployment regression checks. Its comparisons can include output quality, latency, and cost.
What should I provide before evaluating a model?
Set measurable success criteria, a budget, latency and throughput targets, data constraints, and the candidate models under consideration.
