Skip to main content
command CX Codex

analyze-test — Claude Code command: A/B test analysis

analyze-test is a Claude Code command that evaluates A/B test results, checks statistical significance and sample size, and recommends whether to ship, extend, or stop.

MIT ★ 28,211 updated 2026-10-16

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add phuryn/pm-skills
/plugin install pm-data-analytics@pm-skills
Open on GitHub

What analyze-test does

The command ingests test data in any format—conversion rates with group sizes, raw CSV data, screenshots from testing platforms (Optimizely, LaunchDarkly), or written descriptions. It first validates the test design: sufficient sample size, adequate duration to capture business cycles, clean randomization, absence of external noise. Then applies the ab-test-analysis skill to calculate p-value and confidence intervals, absolute and relative lift, practical significance, and segment-level effects.

Output includes a results table with rates and confidence bounds, statistical details (p-value, CI, minimum detectable effect), power analysis verdict, and a decision: SHIP, EXTEND, or STOP. For revenue metrics, it projects business impact at scale and flags concerns: novelty bias, test validity issues, segment divergence.

What it helps with

  • determine statistical significance of an A/B test result
  • validate sample size and test duration are sufficient
  • assess business impact of the observed lift
  • decide whether to ship, extend, or stop the experiment
  • forecast revenue or metric impact if the variant ships to 100%

Example prompts for the agent

  • /analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)
  • /analyze-test [paste CSV data with test results]
  • /analyze-test [screenshot from your experimentation platform]

How to install analyze-test

1Plugin marketplace

/plugin marketplace add phuryn/pm-skills
/plugin install pm-data-analytics@pm-skills

2Manually: put the files in place

.claude/commands/analyze-test.md

Call it with /analyze-test. For all projects use ~/.claude/commands/

Questions about analyze-test

How do I install analyze-test in Claude Code?

Run /plugin marketplace add phuryn/pm-skills, then /plugin install pm-data-analytics@pm-skills, and connect .claude/commands/analyze-test.md.

What does analyze-test do?

It analyzes A/B test results: computes p-value, confidence intervals, checks test power, assesses practical significance, and recommends whether to ship, extend, or stop the experiment.

What data formats does it accept?

Any format works: summary stats (conversion rate and sample size), raw CSV, screenshot from a testing platform, or text description of results.

Source text

analyze-test.md · MIT
---
description: Analyze A/B test results — statistical significance, sample size validation, and ship/extend/stop recommendations
argument-hint: "<test results as data, screenshot, or description>"
---

# /analyze-test -- A/B Test Analysis

Evaluate experiment results with statistical rigor and translate findings into a clear product decision: ship, extend, or stop.

## Invocation

```
/analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)
/analyze-test [upload a CSV of test results]
/analyze-test [screenshot from your experimentation platform]
```

## Workflow

### Step 1: Accept Test Data

Accept in any format:
- Summary statistics (conversion rates, sample sizes per variant)
- Raw event data (CSV with user_id, variant, converted, timestamp)
- Screenshot from an experimentation platform (Optimizely, LaunchDarkly, etc.)
- Description of the experiment and results

### Step 2: Validate Test Design

Before analyzing results, check:
- Was sample size sufficient? (run a power analysis)
- Was the test run long enough? (capture weekly cycles, minimum 1-2 business cycles)
- Was randomization clean? (check for sample ratio mismatch)
- Were there any external factors during the test period?

Flag issues if found — results from a flawed test can be misleading.

### Step 3: Analyze Results

Apply the **ab-test-analysis** skill:

- **Statistical significance**: Calculate p-value and confidence interval
- **Effect size**: Absolute and relative difference between variants
- **Practical significance**: Is the effect large enough to matter for the business?
- **Confidence interval**: What's the range of plausible true effects?
- **Segment analysis**: If data allows, check for differential effects by user segment

### Step 4: Generate Analysis

```
## A/B Test Analysis: [Test Name]

**Date**: [today]
**Test duration**: [X days/weeks]
**Total sample**: [N users]

### Results Summary
| Variant | Sample | Metric | Rate | 95% CI |
|---------|--------|--------|------|--------|
| Control | [n] | [metric] | [X%] | [X% - Y%] |
| Variant | [n] | [metric] | [X%] | [X% - Y%] |

### Statistical Analysis
- **Relative lift**: [+X%] ([CI range])
- **P-value**: [X]
- **Statistically significant**: [Yes/No] at 95% confidence
- **Minimum detectable effect**: [X%] (what the test was powered to detect)

### Sample Size Check
- **Required sample**: [N] per variant (for [X%] MDE at 80% power)
- **Actual sample**: [N] per variant
- **Verdict**: [Sufficiently powered / Underpowered / Overpowered]

### Decision

**Recommendation: [SHIP / EXTEND / STOP]**

[Clear explanation of why, considering both statistical and practical significance]

### Business Impact Estimate
If shipped to 100% of users:
- **Expected impact**: [metric change per month/quarter]
- **Revenue impact**: [if applicable]
- **Confidence**: [How certain we are about this estimate]

### Caveats
- [Any concerns about the test validity]
- [Segments where results differ]
- [Novelty effects or other biases to consider]

### Follow-Up
- [What to test next based on learnings]
- [Monitoring plan if shipping the variant]
```

### Step 5: Offer Next Steps

- "Want me to **design a follow-up experiment** based on these findings?"
- "Should I **run the analysis for specific segments**?"
- "Want me to **generate the SQL** to monitor this metric post-launch?"

## Notes

- Statistical significance ≠ practical significance — a 0.1% lift can be significant with enough data but not worth shipping
- Always check for sample ratio mismatch before trusting results
- Novelty effects can inflate short-term results — recommend monitoring for 2-4 weeks post-launch
- If the test is underpowered, the right answer is usually "extend" not "no effect"
- For revenue metrics, use confidence intervals to estimate best-case and worst-case business impact
- If data is provided as CSV, generate the full analysis using Python with scipy.stats

© Paweł Huryn · MIT · phuryn/pm-skills

Similar slash commands

All slash commands for Testing and QA →

From the same set120

The whole set →
pre-mortem The pre-mortem command stress-tests a PRD, launch plan, or feature before release by finding failure risks and turning them into follow-up actions. CX Codex +1 ★ 28,211repo stars stakeholder-map stakeholder-map — a command that maps stakeholders on a Power × Interest grid and builds a communication plan. Use it when a project affects different groups and you need to ensure each receives the right information at the right time. CX Codex +1 ★ 28,211repo stars battlecard Generates a concise, sales-focused competitive battlecard comparing two products. Useful before prospect calls to align pricing rebuttals and differentiators. CX Codex +1 ★ 28,211repo stars write-stories write-stories is a Claude Code command that turns a feature brief or PRD into backlog items with acceptance criteria. Choose user stories, job stories, or WWA. CX Codex +1 ★ 28,211repo stars north-star Identifies your North Star Metric, validates it against core criteria, and outlines actionable input metrics. CX Codex +1 ★ 28,211repo stars growth-strategy growth-strategy builds a growth plan from growth loops and GTM motions. Use it when product traction stalls or you need a structured experiment plan. CX Codex +1 ★ 28,211repo stars strategy The /strategy command builds a complete product strategy through a 9-section Canvas: vision, segments, value, trade-offs, metrics, growth, capabilities, and defensibility. CX Codex +1 ★ 28,211repo stars research-users research-users synthesizes research notes, survey exports, and tickets into user personas, behavioral segments, and journey maps. Helpful for PMs evaluating qualitative data. CX Codex +1 ★ 28,211repo stars

Project sponsors

Companies that keep this analytics open