Skip to main content
subagent CX Codex

eval-engineer — Codex subagent: what it does, how to install

eval-engineer — a Codex subagent that designs evaluation setups for prompts, retrieval pipelines, tool calling, and multi-step agents.

MIT ★ 6,505 updated 2026-10-05

Install into Codex · Manually: put the files in place All methods and platforms ↓
.codex/agents/eval-engineer.toml
Open on GitHub

What eval-engineer does

The subagent structures measurement engineering for production setups rather than running vanity benchmarks. It maps out the target workflow, spots failure risks, and turns them into test scenarios. You get a prioritized scenario matrix, scoring rubrics, regression thresholds, and impact tracking for latency and token cost.

Its instructions bar declaring an evaluation complete when it merely covers the happy path. The agent explicitly differentiates between automated offline checks, human review, and live production validation. The default configuration targets the gpt-5.6-sol model.

What it helps with

  • design an evaluation framework for multi-step agent pipelines
  • establish regression thresholds for retrieval and tool-use workflows
  • identify edge cases and failure modes for complex prompts
  • separate offline automated checks from required human expert review

Example prompts for the agent

  • Design an evaluation matrix for our customer support retrieval pipeline
  • Define regression criteria and test scenarios for a database tool-calling agent
  • Build an evaluation plan to compare model changes across multi-turn dialogs

How to install eval-engineer

Manually: put the files in place

.codex/agents/eval-engineer.toml

Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about eval-engineer

How to install eval-engineer in Codex?

Place the subagent configuration into .codex/agents/eval-engineer.toml in your repository.

What does the eval-engineer subagent do?

It builds evaluation designs with concrete failure scenarios, metric definitions, regression guardrails, and go/no-go thresholds.

Which model does eval-engineer specify?

The subagent frontmatter is configured to run on gpt-5.6-sol.

Source text

eval-engineer.toml · MIT
# eval-engineer

Use when a task needs evaluation design for prompts, retrieval, tools, or multi-step agent workflows.

Own evaluation design as measurement engineering for real system quality, not vanity benchmarking.

Working mode:
1. Define the workflow under test and the decisions the evaluation should support.
2. Identify the highest-risk failure modes and translate them into measurable scenarios.
3. Build the leanest useful evaluation plan that can catch regressions and compare changes.
4. Distinguish offline evaluation, human review, and live validation needs.

Focus on:
- scenario coverage tied to real tasks and edge cases
- pass/fail criteria, rubrics, and judgment consistency
- retrieval, tool-use, and multi-turn workflow failure measurement
- cost and latency impacts alongside output quality
- regression thresholds that are strict enough to matter

Quality checks:
- ensure the eval plan can influence actual go/no-go decisions
- avoid proxy metrics that hide real user failures
- separate dataset gaps from model or workflow failures
- call out where human labels or expert review are necessary

Return:
- evaluation objective and target workflow
- prioritized scenario matrix and metrics
- scoring or review approach
- regression strategy and decision thresholds
- limitations and what still requires live testing

Do not claim an evaluation is comprehensive when it only samples a narrow happy path unless explicitly requested by the parent agent.

© VoltAgent · MIT · VoltAgent/awesome-codex-subagents

Similar subagents

All subagents for Testing and QA →
eval-runner haabe eval-runner — a skill for Claude Code, Codex, Cursor and other agents that runs YAML eval scenarios through a reflexion loop and writes results to JSON. CX Codex +1 ★ 54repo stars bmad-eval bmad-code-org bmad-eval — a skill for Claude Code, Codex, Cursor and Copilot that runs another skill's test suite and reports findings. Triggers when you need to check description accuracy, compare versions or grade output against a rubric. CX Codex +1 9,100installs eval-harness affaan-m eval-harness — a skill for Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, and OpenCode that brings eval-driven development into every agent session: success criteria go in before any code, results are scored with pass@k and pass^k. CX Codex +1 6,500installs eval-engineering langchain-ai eval-engineering is a Claude Code skill for designing and checking tasks that evaluate AI agents. It comes into play with benchmark work, run traces, and Harbor tasks. CX Codex +1 6,300installs agent-observability-eval-pipeline datadog-labs agent-observability-eval-pipeline walks an instrumented ml_app through six phases: classify traces, root-cause failures, bootstrap evaluators, build a dataset, run an experiment, analyze it. CX Codex +1 674installs llm-eval-harness daymade llm-eval-harness — a skill for Claude Code, Codex, Cursor, and other platforms that probes a third-party LLM endpoint for availability, speed, system-prompt fidelity, and protocol correctness. CX Codex +1 438installs

From the same set176

The whole set →
ponder-reviewer ponder-reviewer — a subagent for Codex runs a coverage-gated, multi-pass deliberative review protocol with false-positive retraction. CX Codex ★ 6,505repo stars prompt-regression-tester prompt-regression-tester is a Codex subagent that builds regression coverage for changes to AI prompts, models, tools, or workflows. It identifies likely behavior shifts and suggests test criteria. CX Codex ★ 6,505repo stars hallucination-investigator hallucination-investigator is a Codex subagent that diagnoses why AI outputs fail, tracing errors to context, retrieval, prompts, tools, or unsupported inference. CX Codex ★ 6,505repo stars ai-observability-engineer ai-observability-engineer is a Codex subagent that designs telemetry, metrics, and tracing for production LLM pipelines. It steps in when probabilistic workflows require visibility. CX Codex ★ 6,505repo stars platform-product-manager platform-product-manager — a Codex subagent for internal platform work: it turns team pain points into a roadmap with adoption metrics and stated tradeoffs. CX Codex ★ 6,505repo stars idp-architect idp-architect designs internal developer platform architectures, service boundaries, and self-service control planes for engineering teams. CX Codex ★ 6,505repo stars golden-path-designer golden-path-designer designs a standard route for creating, deploying, and operating services. Use it when a team needs sensible defaults and clear boundaries for exceptions. CX Codex ★ 6,505repo stars backstage-specialist backstage-specialist plans an internal developer portal on Backstage: service catalog, scaffolder templates, plugin choices and a rollout path. CX Codex ★ 6,505repo stars

Project sponsors

Companies that keep this analytics open