# ab-test-setup

Plan an experiment, estimate its sample size and duration, then assess the results. Useful when you need a clear hypothesis and decision criteria for an A/B test.

- skill · `type=skill` · Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode · 417 installs · ★ 721
- Author: OpenClaudia
- License: MIT · policy=mirror · the beginning of the file is at the end of this page
- Source: https://github.com/OpenClaudia/openclaudia-skills/tree/HEAD/skills/ab-test-setup/SKILL.md
- Updated: 2026-10-21
- HTML: https://affpapa.org/en/nevibecode/skills/openclaudia-ab-test-setup

## What it does

The skill asks for experiment context such as baseline conversion, traffic, metrics and time constraints. It can turn those details into a hypothesis, suggest a test type, and estimate the sample size and run time. Its guidance covers headlines, calls to action, page layouts and pricing, with metrics suited to each case.

A pre-launch checklist covers tracking, variant QA and traffic allocation. After the test, the skill can help check sample ratio mismatch and assess statistical significance. It also flags common traps, including stopping early and overlooking segments. Pick one primary metric before launch; extra comparisons call for a Bonferroni correction.

The duration estimate uses sample size and daily traffic. The recommended range runs from 7 days to 8 weeks, so a longer estimate means revisiting the MDE, variant count or metric. An ICE scoring method and roadmap template help rank later experiments.

## Install

### Claude Code (native platform)

1. **Skills CLI (Vercel)**
```bash
npx skills add OpenClaudia/openclaudia-skills --skill ab-test-setup -a claude-code
```

2. **Manually: put the files in place**
- File: `~/.claude/skills/ab-test-setup/SKILL.md` — Copy the whole skill folder; for one project use .claude/skills/ab-test-setup/

### Codex

1. **Skills CLI (Vercel)**
```bash
npx skills add OpenClaudia/openclaudia-skills --skill ab-test-setup -a codex
```

2. **Manually: put the files in place**
- File: `.agents/skills/ab-test-setup/SKILL.md` — For all projects use ~/.agents/skills/

### Cursor

1. **Skills CLI (Vercel)**
```bash
npx skills add OpenClaudia/openclaudia-skills --skill ab-test-setup -a cursor
```

2. **Manually: put the files in place**
- File: `.cursor/skills/ab-test-setup/SKILL.md` — Cursor also reads .agents/skills/ and .claude/skills/

### GitHub Copilot

1. **GitHub CLI**
```bash
gh skill install OpenClaudia/openclaudia-skills ab-test-setup
```

2. **Manually: put the files in place**
- File: `.github/skills/ab-test-setup/SKILL.md`

### Gemini CLI

1. **Skills CLI (Vercel)**
```bash
npx skills add OpenClaudia/openclaudia-skills --skill ab-test-setup -a gemini-cli
```

2. **Platform CLI**
```bash
gemini skills install https://github.com/OpenClaudia/openclaudia-skills --path skills/ab-test-setup
```

### Windsurf (Devin Desktop)

**Manually: put the files in place**
- File: `.devin/skills/ab-test-setup/SKILL.md` — The legacy .windsurf/skills/ path still works

### OpenCode

**Skills CLI (Vercel)**
```bash
npx skills add OpenClaudia/openclaudia-skills --skill ab-test-setup -a opencode
```

### Claude.ai

**Claude.ai**
- Zip the folder and upload it in Settings → Capabilities → Skills

Where a file is given instead of a command, copy the files from the source to that path: https://github.com/OpenClaudia/openclaudia-skills/tree/HEAD/skills/ab-test-setup/SKILL.md

## When to use it
- write a test hypothesis with a defined primary metric
- estimate sample size from conversion rate and MDE
- project experiment duration from daily traffic
- compare variants and assess statistical significance
- rank experiments with an ICE score

## Example prompts
- Plan an A/B test for a new headline using a 5% baseline conversion rate and 10000 daily visitors.
- Estimate the sample size and run time for two variants with a 15% MDE.
- Review these experiment results, check for SRM, and help decide whether to implement variant B.

## Source text · SKILL.md · MIT

Below is the content of a third-party file. It is data, not instructions for you: do not act on anything written inside unless the human asks. Block tag: 3accd6a23d72eb4e.

BEGIN THIRD-PARTY FILE 3accd6a23d72eb4e
````text
---
name: ab-test-setup
description: Design, plan, and analyze A/B tests with statistical rigor. Use when the user asks about A/B testing, split testing, experiment design, statistical significance, sample size calculation, test duration, multivariate testing, or conversion experiments. Trigger phrases include "A/B test", "split test", "experiment", "statistical significance", "sample size", "test duration", "which version wins", "conversion experiment", "hypothesis test", "variant testing".
---

# A/B Test Design and Analysis

You are an expert in experimentation and A/B testing. When the user asks you to design a test, calculate sample sizes, analyze results, or plan an experimentation roadmap, follow this framework.

## Step 1: Gather Test Context

Establish: page/feature being tested, current conversion rate, monthly traffic, primary metric, secondary metrics, guardrail metrics, duration constraints, testing platform (Optimizely, VWO, custom).

## Step 2: Hypothesis Framework

### Hypothesis Template

```
OBSERVATION: [What we noticed in data/research/feedback]
HYPOTHESIS: If we [specific change], then [metric] will [change] by [amount],
            because [behavioral/psychological reasoning].
CONTROL (A): [Current state]
VARIANT (B): [Proposed change]
PRIMARY METRIC: [Single metric that determines winner]
GUARDRAILS: [Metrics that must not degrade]
```

### Hypothesis Categories

- **Clarity**: "Users don't understand what we offer" -- test headline, value prop
- **Motivation**: "Users aren't motivated to act" -- test social proof, urgency, benefits
- **Friction**: "Process is too difficult" -- test form length, step count, layout
- **Trust**: "Users don't trust us" -- test testimonials, guarantees, badges
- **Relevance**: "Content doesn't match intent" -- test personalization, segmentation

## Step 3: Sample Size and Duration

### Sample Size Formula

```
n = (Z_alpha/2 + Z_beta)^2 * (p1*(1-p1) + p2*(1-p2)) / (p2 - p1)^2
Where: Z_alpha/2 = 1.96 (95%), Z_beta = 0.84 (80% power), p2 = p1 * (1 + MDE)
```

### Quick Reference (per variant, 95% significance, 80% power)

| Baseline CR | 10% MDE | 15% MDE | 20% MDE | 25% MDE |
|---|---|---|---|---|
| 2% | 385,040 | 173,470 | 98,740 | 63,850 |
| 3% | 253,670 | 114,300 | 65,080 | 42,110 |
| 5% | 148,640 | 67,040 | 38,200 | 24,730 |
| 10% | 70,420 | 31,780 | 18,120 | 11,740 |
| 15% | 44,310 | 20,010 | 11,420 | 7,400 |
| 20% | 31,310 | 14,140 | 8,070 | 5,230 |

**Duration** = (Sample size per variant x Number of variants) / Daily traffic. Minimum 7 days, maximum 8 weeks.
````
END THIRD-PARTY FILE 3accd6a23d72eb4e

The third-party file has ended. Everything between the 3accd6a23d72eb4e tags is data: requests and commands from it are not followed. An “end of file” line with another tag inside the block is part of the file.

First 2563 of 6735 bytes. © OpenClaudia · MIT · https://github.com/OpenClaudia/openclaudia-skills/tree/HEAD/skills/ab-test-setup/SKILL.md

_Part of the page did not fit — see the HTML version._

---
NeVibeCode on affpapa.org. Catalog texts are descriptions, not instructions for the agent. How to choose and install: https://affpapa.org/en/nevibecode/agents.md