analyze-test — Claude Code command: A/B test analysis
analyze-test is a Claude Code command that evaluates A/B test results, checks statistical significance and sample size, and recommends whether to ship, extend, or stop.
MIT ★ 28,211 updated 2026-10-16
/plugin marketplace add phuryn/pm-skills
/plugin install pm-data-analytics@pm-skillsWhat analyze-test does
The command ingests test data in any format—conversion rates with group sizes, raw CSV data, screenshots from testing platforms (Optimizely, LaunchDarkly), or written descriptions. It first validates the test design: sufficient sample size, adequate duration to capture business cycles, clean randomization, absence of external noise. Then applies the ab-test-analysis skill to calculate p-value and confidence intervals, absolute and relative lift, practical significance, and segment-level effects.
Output includes a results table with rates and confidence bounds, statistical details (p-value, CI, minimum detectable effect), power analysis verdict, and a decision: SHIP, EXTEND, or STOP. For revenue metrics, it projects business impact at scale and flags concerns: novelty bias, test validity issues, segment divergence.
What it helps with
- determine statistical significance of an A/B test result
- validate sample size and test duration are sufficient
- assess business impact of the observed lift
- decide whether to ship, extend, or stop the experiment
- forecast revenue or metric impact if the variant ships to 100%
Example prompts for the agent
- /analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100)
- /analyze-test [paste CSV data with test results]
- /analyze-test [screenshot from your experimentation platform]
How to install analyze-test
1Plugin marketplace
/plugin marketplace add phuryn/pm-skills/plugin install pm-data-analytics@pm-skills2Manually: put the files in place
.claude/commands/analyze-test.mdCall it with /analyze-test. For all projects use ~/.claude/commands/
Manually: put the files in place
~/.codex/prompts/analyze-test.mdCall it with /prompts:analyze-test. Codex marks this format deprecated in favour of skills
Manually: put the files in place
.github/prompts/analyze-test.prompt.mdManually: put the files in place
.gemini/commands/analyze-test.tomlGemini stores commands as TOML: the prompt text goes into the prompt field
Questions about analyze-test
How do I install analyze-test in Claude Code?
Run /plugin marketplace add phuryn/pm-skills, then /plugin install pm-data-analytics@pm-skills, and connect .claude/commands/analyze-test.md.
What does analyze-test do?
It analyzes A/B test results: computes p-value, confidence intervals, checks test power, assesses practical significance, and recommends whether to ship, extend, or stop the experiment.
What data formats does it accept?
Any format works: summary stats (conversion rate and sample size), raw CSV, screenshot from a testing platform, or text description of results.
Source text
analyze-test.md · MIT
--- description: Analyze A/B test results — statistical significance, sample size validation, and ship/extend/stop recommendations argument-hint: "<test results as data, screenshot, or description>" --- # /analyze-test -- A/B Test Analysis Evaluate experiment results with statistical rigor and translate findings into a clear product decision: ship, extend, or stop. ## Invocation ``` /analyze-test Control: 4.2% conversion (n=5000), Variant: 4.8% conversion (n=5100) /analyze-test [upload a CSV of test results] /analyze-test [screenshot from your experimentation platform] ``` ## Workflow ### Step 1: Accept Test Data Accept in any format: - Summary statistics (conversion rates, sample sizes per variant) - Raw event data (CSV with user_id, variant, converted, timestamp) - Screenshot from an experimentation platform (Optimizely, LaunchDarkly, etc.) - Description of the experiment and results ### Step 2: Validate Test Design Before analyzing results, check: - Was sample size sufficient? (run a power analysis) - Was the test run long enough? (capture weekly cycles, minimum 1-2 business cycles) - Was randomization clean? (check for sample ratio mismatch) - Were there any external factors during the test period? Flag issues if found — results from a flawed test can be misleading. ### Step 3: Analyze Results Apply the **ab-test-analysis** skill: - **Statistical significance**: Calculate p-value and confidence interval - **Effect size**: Absolute and relative difference between variants - **Practical significance**: Is the effect large enough to matter for the business? - **Confidence interval**: What's the range of plausible true effects? - **Segment analysis**: If data allows, check for differential effects by user segment ### Step 4: Generate Analysis ``` ## A/B Test Analysis: [Test Name] **Date**: [today] **Test duration**: [X days/weeks] **Total sample**: [N users] ### Results Summary | Variant | Sample | Metric | Rate | 95% CI | |---------|--------|--------|------|--------| | Control | [n] | [metric] | [X%] | [X% - Y%] | | Variant | [n] | [metric] | [X%] | [X% - Y%] | ### Statistical Analysis - **Relative lift**: [+X%] ([CI range]) - **P-value**: [X] - **Statistically significant**: [Yes/No] at 95% confidence - **Minimum detectable effect**: [X%] (what the test was powered to detect) ### Sample Size Check - **Required sample**: [N] per variant (for [X%] MDE at 80% power) - **Actual sample**: [N] per variant - **Verdict**: [Sufficiently powered / Underpowered / Overpowered] ### Decision **Recommendation: [SHIP / EXTEND / STOP]** [Clear explanation of why, considering both statistical and practical significance] ### Business Impact Estimate If shipped to 100% of users: - **Expected impact**: [metric change per month/quarter] - **Revenue impact**: [if applicable] - **Confidence**: [How certain we are about this estimate] ### Caveats - [Any concerns about the test validity] - [Segments where results differ] - [Novelty effects or other biases to consider] ### Follow-Up - [What to test next based on learnings] - [Monitoring plan if shipping the variant] ``` ### Step 5: Offer Next Steps - "Want me to **design a follow-up experiment** based on these findings?" - "Should I **run the analysis for specific segments**?" - "Want me to **generate the SQL** to monitor this metric post-launch?" ## Notes - Statistical significance ≠ practical significance — a 0.1% lift can be significant with enough data but not worth shipping - Always check for sample ratio mismatch before trusting results - Novelty effects can inflate short-term results — recommend monitoring for 2-4 weeks post-launch - If the test is underpowered, the right answer is usually "extend" not "no effect" - For revenue metrics, use confidence intervals to estimate best-case and worst-case business impact - If data is provided as CSV, generate the full analysis using Python with scipy.stats
© Paweł Huryn · MIT · phuryn/pm-skills