Skip to main content
subagent CX Codex

ab-test-analysis · voltagent/awesome-codex-subagents — Codex subagent: what it does, how to install

ab-test-analysis evaluates A/B test data, inspects statistical and practical significance, and delivers principled ship or no-ship decisions.

MIT ★ 6,505 updated 2026-10-05

Install into Codex · Manually: put the files in place All methods and platforms ↓
.codex/agents/ab-test-analysis.toml
Open on GitHub

What ab-test-analysis does

This subagent audits experiment outcomes instead of treating decisions as simple p-value lookups. It checks experiment design from the start: sample ratio mismatch, statistical power, duration, and peeking traps.

Analysis focuses on effect sizes, confidence intervals, and guardrail metrics. The agent flags distortions like Simpson's paradox, survivorship bias, and post-hoc segmentation. A primary metric gain will not trigger a ship verdict if it harms a guardrail metric.

Each run returns a structured summary table alongside an explicit recommendation: ship, no-ship, iterate, or inconclusive.

What it helps with

  • Validate experiment integrity against sample ratio mismatch
  • Separate statistical significance from practical effect size
  • Verify guardrail metrics before recommending feature release
  • Issue an explicit ship, no-ship, or iterate verdict

Example prompts for the agent

  • Analyze our checkout A/B test results from the provided variant metrics
  • Check whether sample ratio mismatch or peeking invalidates this experiment
  • Evaluate lift, confidence intervals, and guardrail metrics to issue a ship verdict

How to install ab-test-analysis

Manually: put the files in place

.codex/agents/ab-test-analysis.toml

Codex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/

Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean

Questions about ab-test-analysis

How to install ab-test-analysis in Codex?

Add the subagent definition file to .codex/agents/ab-test-analysis.toml.

What does ab-test-analysis do?

It audits A/B test data for statistical validity, checks guardrails, and provides an actionable decision on whether to launch the change.

Why does ab-test-analysis reject wins based solely on p-values?

Statistical significance alone does not prove practical impact and can obscure damage to critical guardrail metrics.

Source text

ab-test-analysis.toml · MIT
# ab-test-analysis

Use when a task needs analysis of A/B test results, interpretation of p-values and confidence intervals, statistical significance checks, or a principled ship/no-ship decision.

Own A/B test analysis as a principled ship/no-ship decision, not a p-value lookup.

Separate statistical significance from practical significance, and catch common analysis traps before recommending action.

Working mode:
1. Establish the experiment design: primary metric, guardrails, pre-specified minimum detectable effect, and duration.
2. Validate test integrity (sample ratio match, sufficient power, no peeking) before trusting any result.
3. Quantify the effect: observed lift, confidence interval, p-value, and whether the result is practically meaningful.
4. Issue a ship / no-ship / iterate / inconclusive verdict with explicit rationale.

Focus on:
- p-value meaning: probability of data this extreme under the null, not the probability the variant is better
- effect size and confidence interval as the decision drivers, not significance alone
- guardrail metrics: never ship a primary win that significantly harms a guardrail
- pre-specified segments only; flag post-hoc segmentation as p-hacking
- common errors: peeking, multiple comparisons, Simpson's paradox, survivorship bias
- frequentist vs bayesian framing and which the tooling actually used
- power and sample adequacy when results are trending but underpowered

Quality checks:
- confirm the verdict aligns with primary metric, effect size, and guardrail status together
- verify no sample ratio mismatch invalidates the test
- check the test ran to its predetermined sample size and duration
- ensure each segment claim was pre-planned, not mined after the fact
- distinguish "no effect detected" from "underpowered, extend the test"

Return:
- results summary table (control vs treatment: n, conversion rate, lift, CI, p-value)
- statistical significance verdict and effect-size interpretation
- guardrail metric status
- ship / no-ship / iterate / inconclusive recommendation with rationale
- next step (ship plan, extension, or learning to capture)

Do not declare a win on statistical significance alone when practical significance or guardrails are unmet unless explicitly requested by the parent agent.

© VoltAgent · MIT · VoltAgent/awesome-codex-subagents

Similar subagents

All subagents for Data and databases →

ab-test-analysis by other authors comes first.

ab-test-analysis VoltAgent Community Subagent for A/B test analysis: interprets p-values, calculates confidence intervals, and helps decide whether to ship a change to production. CX Codex ★ 26,718repo stars ad-test-designer aaron-he-zhu ad-test-designer plans creative and landing-page experiments, then reads results from your exported data. It also helps set up incrementality tests. CX Codex +1 1,100installs analyze-test Paweł Huryn analyze-test is a Claude Code command that evaluates A/B test results, checks statistical significance and sample size, and recommends whether to ship, extend, or stop. CX Codex +1 ★ 28,211repo stars ab-test-setup OpenClaudia ab-test-setup is a Claude Code skill for designing A/B tests, sizing samples, and reviewing results. It activates for requests about experiments, significance, or conversion tests. CX Codex +1 435installs power-analysis gaasher power-analysis is a Claude Code skill for sizing a two-group study and drafting its preregistration. It runs during planning, before data collection, for A/B tests, simple RCTs, and evaluations of two models or configurations. CX Codex +1 ★ 185repo stars ads-test AgriciDaniel ads-test is a skill for Claude Code, Codex, Cursor and other platforms that guides paid-media experiment design and readout, from hypothesis to decision. CX Codex +1 4,000installs experimentation-analytics rampstackco experimentation-analytics — a skill for Claude Code, Codex, Cursor, Copilot, and Gemini CLI that walks through a finished experiment result panel before a ship-or-kill call. CX Codex +1 283installs

From the same set176

The whole set →
docs-researcher docs-researcher — a subagent that verifies APIs, version-specific behavior and framework options against official documentation. CX Codex ★ 6,505repo stars data-researcher The data-researcher subagent for Codex gathers and synthesizes datasets, metrics, and quantitative evidence for engineering and product decisions, rather than dumping raw links. CX Codex ★ 6,505repo stars competitive-analyst competitive-analyst compares tools, products, libraries, and implementation options against criteria for a specific decision. Useful when adoption and maintenance matter as much as features. CX Codex ★ 6,505repo stars cohort-analysis cohort-analysis — a subagent for diagnosing product health via retention curves and user cohort behavior. Separates product-market-fit problems from growth issues and identifies activation moments that predict long-term engagement. CX Codex ★ 6,505repo stars workflow-orchestrator The workflow-orchestrator subagent turns a complex request into staged Codex work, with delegation rules, validation gates, integration steps, and failure handling. CX Codex ★ 6,505repo stars task-distributor task-distributor is a Codex subagent that breaks broad tasks into work units with boundaries, dependencies, and ownership. It divides work across agents without duplication or overlap. CX Codex ★ 6,505repo stars performance-monitor performance-monitor triages build, runtime, and operational metrics, weeds out noise, and pinpoints responsible subsystems before heavy profiling starts. CX Codex ★ 6,505repo stars multi-agent-coordinator The multi-agent-coordinator subagent divides work between the parent agent and parallel delegates, sets ownership boundaries, and plans how results come together. Use it for tasks with separable workstreams. CX Codex ★ 6,505repo stars

Project sponsors

Companies that keep this analytics open