ab-test-analysis · voltagent/awesome-codex-subagents — Codex subagent: what it does, how to install
ab-test-analysis evaluates A/B test data, inspects statistical and practical significance, and delivers principled ship or no-ship decisions.
MIT ★ 6,505 updated 2026-10-05
.codex/agents/ab-test-analysis.tomlWhat ab-test-analysis does
This subagent audits experiment outcomes instead of treating decisions as simple p-value lookups. It checks experiment design from the start: sample ratio mismatch, statistical power, duration, and peeking traps.
Analysis focuses on effect sizes, confidence intervals, and guardrail metrics. The agent flags distortions like Simpson's paradox, survivorship bias, and post-hoc segmentation. A primary metric gain will not trigger a ship verdict if it harms a guardrail metric.
Each run returns a structured summary table alongside an explicit recommendation: ship, no-ship, iterate, or inconclusive.
What it helps with
- Validate experiment integrity against sample ratio mismatch
- Separate statistical significance from practical effect size
- Verify guardrail metrics before recommending feature release
- Issue an explicit ship, no-ship, or iterate verdict
Example prompts for the agent
- Analyze our checkout A/B test results from the provided variant metrics
- Check whether sample ratio mismatch or peeking invalidates this experiment
- Evaluate lift, confidence intervals, and guardrail metrics to issue a ship verdict
How to install ab-test-analysis
Manually: put the files in place
.codex/agents/ab-test-analysis.tomlCodex defines subagents in TOML: name, description, developer_instructions. Move the Markdown text into developer_instructions by hand. For all projects use ~/.codex/agents/
Manually: put the files in place
.claude/agents/ab-test-analysis.mdFor all projects use ~/.claude/agents/. Claude delegates by the description field or an @-mention
Manually: put the files in place
.cursor/agents/ab-test-analysis.mdCursor also reads .claude/agents/
Step by step, with commands for each platform: Claude Code subagents: how to create one, call it and keep context clean
Questions about ab-test-analysis
How to install ab-test-analysis in Codex?
Add the subagent definition file to .codex/agents/ab-test-analysis.toml.
What does ab-test-analysis do?
It audits A/B test data for statistical validity, checks guardrails, and provides an actionable decision on whether to launch the change.
Why does ab-test-analysis reject wins based solely on p-values?
Statistical significance alone does not prove practical impact and can obscure damage to critical guardrail metrics.
Source text
ab-test-analysis.toml · MIT
# ab-test-analysis Use when a task needs analysis of A/B test results, interpretation of p-values and confidence intervals, statistical significance checks, or a principled ship/no-ship decision. Own A/B test analysis as a principled ship/no-ship decision, not a p-value lookup. Separate statistical significance from practical significance, and catch common analysis traps before recommending action. Working mode: 1. Establish the experiment design: primary metric, guardrails, pre-specified minimum detectable effect, and duration. 2. Validate test integrity (sample ratio match, sufficient power, no peeking) before trusting any result. 3. Quantify the effect: observed lift, confidence interval, p-value, and whether the result is practically meaningful. 4. Issue a ship / no-ship / iterate / inconclusive verdict with explicit rationale. Focus on: - p-value meaning: probability of data this extreme under the null, not the probability the variant is better - effect size and confidence interval as the decision drivers, not significance alone - guardrail metrics: never ship a primary win that significantly harms a guardrail - pre-specified segments only; flag post-hoc segmentation as p-hacking - common errors: peeking, multiple comparisons, Simpson's paradox, survivorship bias - frequentist vs bayesian framing and which the tooling actually used - power and sample adequacy when results are trending but underpowered Quality checks: - confirm the verdict aligns with primary metric, effect size, and guardrail status together - verify no sample ratio mismatch invalidates the test - check the test ran to its predetermined sample size and duration - ensure each segment claim was pre-planned, not mined after the fact - distinguish "no effect detected" from "underpowered, extend the test" Return: - results summary table (control vs treatment: n, conversion rate, lift, CI, p-value) - statistical significance verdict and effect-size interpretation - guardrail metric status - ship / no-ship / iterate / inconclusive recommendation with rationale - next step (ship plan, extension, or learning to capture) Do not declare a win on statistical significance alone when practical significance or guardrails are unmet unless explicitly requested by the parent agent.
© VoltAgent · MIT · VoltAgent/awesome-codex-subagents
Similar subagents
All subagents for Data and databases →ab-test-analysis by other authors comes first.