Skip to main content
skill CX Codex

power-analysis · gaasher — Claude Code skill: sample sizing and setup

power-analysis is a Claude Code skill for sizing a two-group study and drafting its preregistration. It runs during planning, before data collection, for A/B tests, simple RCTs, and evaluations of two models or configurations.

MIT ★ 185 updated 2026-06-30

Install into Claude Code · Plugin marketplace All methods and platforms ↓
/plugin marketplace add gaasher/Agent-Loop-Skills
/plugin install agent-loops@agent-loop-skills
Open on GitHub

What power-analysis does

power-analysis simulates statistical power to find the per-group sample size for a stated minimum effect. It also checks the study plan against a validity checklist and prepares a sample-size justification with a preregistration covering the hypothesis, primary outcome, analysis, and stopping rule.

The skill handles one two-arm comparison with one primary outcome: a continuous mean or a binary rate. Its simulation uses tools/power_sim.py, which requires Python 3.9 or later. Factorial, repeated-measures, clustered, time-series, adaptive, and survival designs fall outside its scope, as does analysis of data already collected.

What it helps with

  • size a two-group study for a target statistical power
  • audit a study plan against a validity checklist
  • preregister the hypothesis, primary outcome, and stopping rule before collecting data

Example prompts for the agent

  • Size an A/B test for a continuous primary outcome and draft its preregistration.
  • Audit a two-model comparison with a binary outcome against the validity checklist.

How to install power-analysis

1Plugin marketplace

/plugin marketplace add gaasher/Agent-Loop-Skills
/plugin install agent-loops@agent-loop-skills

This installs the whole agent-loops plugin with all its skills

2Skills CLI (Vercel)

npx skills add gaasher/Agent-Loop-Skills --skill power-analysis -a claude-code

3Manually: put the files in place

~/.claude/skills/power-analysis/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/power-analysis/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about power-analysis

How do I install power-analysis in Claude Code?

Add the marketplace with /plugin marketplace add gaasher/Agent-Loop-Skills, then install the plugin with /plugin install agent-loops@agent-loop-skills.

What does power-analysis do?

It simulates power for a comparison between two groups, finds the required group size, checks the study plan, and drafts a preregistration before data collection.

What does power-analysis need to run?

Python 3.9 or later is required. The skill covers a single two-arm comparison with one continuous or binary primary outcome.

Source text

SKILL.md · MIT
---
name: power-analysis
description: >
  Use when the user is planning a two-arm comparison (an A/B test, a simple RCT, a behavioral study, or
  a two-model/two-config evaluation) and needs to size it and preregister it before collecting data —
  finding the per-group sample size that hits target statistical power for the smallest effect worth
  detecting, auditing the design against a validity checklist, and locking it in a preregistration.
  Only for a single two-arm comparison with one primary outcome. Not for factorial, repeated-measures,
  clustered/multilevel, time-series, adaptive, or survival designs; not for analyzing data already
  collected; not for choosing the outcome or manipulation from domain knowledge.
compatibility: Requires Python 3.9+
metadata:
  version: "0.1.0"
---

# Power Analysis Loop

A **power-analysis-and-preregister** loop for a **two-arm comparison**. The artifact is the study's
statistical plan; the feedback signal is two parts — **statistical power** (estimated by Monte-Carlo
simulation of the planned test) and a **count of validity flaws**. Each iteration simulates power,
solves for the sample size that reaches the target, audits the design for flaws, and revises — until
power clears the target **and** the flaw list is empty. The deliverable is a sample-size justification
plus a preregistration that pins the hypothesis, primary outcome, analysis, sample size, and stopping
rule before any data is seen.

## Scope & limitations
This loop does exactly three things, in a loop: **(1)** computes power and required sample size for a
**two-group comparison** by simulation, **(2)** runs a fixed **validity checklist** over the design,
and **(3)** writes a **preregistration**. The vendored power model (`tools/power_sim.py`) covers
**two-sample mean** (continuous outcome) and **two-proportion** (binary outcome) tests only.

It is **not** a general experiment designer. It does **not** handle factorial, repeated-measures,
clustered/multilevel, time-series, adaptive, or survival designs; it does **not** pick your outcome
measure or manipulation from domain knowledge; and it does **not** analyze data you have already
collected. For those, the power numbers here do not apply — use a design-appropriate power method. If
the study is not a simple two-arm comparison, say so and stop rather than reporting a power that does
not match the planned analysis.

## When to use
Use this to size and preregister one two-arm comparison whose primary outcome is a continuous mean or a
binary rate. Default to powering for the **minimal effect of interest** the user states; if they are
unsure of that effect, help them set it from a baseline and a smallest-meaningful difference rather than
an optimistic guess — a design "powered" for an effect bigger than reality is a fiction. If the study is
not a two-arm comparison, stop and point to a design-appropriate method.

## Setup
Resolve bindings interactively. If `loop.run.yaml` exists in the working dir, load it, confirm the
values in one line, and skip to the loop. Otherwise: on Claude Code (the `AskUserQuestion` tool is
available) infer a likely value for each binding and present it as the recommended option; on other
hosts ask each as a quoted plain-text prompt. Then write `loop.run.yaml` (format:
`examples/run.example.yaml`) and confirm the values before creating any other files.

| binding | meaning | default | how to infer |
|---|---|---|---|
| `<hypothesis>` | the claim the experiment tests | — | ask the user |
| `<outcome>` | primary outcome type + minimal effect of interest: continuous (`baseline_mean`, `sd`, `min_effect`) **or** binary (`baseline_rate`, `min_lift`) | — | ask; this fixes the effect size power is computed at |
| `<target_power>` | power the design must clear | `0.80` | — |
| `<alpha>` | significance level | `0.05` | — |
| `<power_cmd>` | invocation of the vendored simulator | `python3 <skill_dir>/tools/power_sim.py --design <two-sample-mean\|two-proportion> --effect <e> [--sd <sd> \| --baseline <p0>] --alpha <alpha> --n <n_per_group>` | — |
| `<design_doc>` | output design + preregistration file | `<sandbox_root>/design.md` | — |
| `<sandbox_root>` | where design + ledger live | `./sandbox` | — |
| `<budget>` | max iterations | 8 | — |

`<power_cmd>` prints one JSON object, `{"power", "n_per_group", ...}`. **Run** it to get the power; never
estimate power by hand.

## The loop
Copy this checklist and tick items off:
- [ ] Iteration 0 — draft the design to `<design_doc>`; record nothing as final.
- [ ] **Simulate** power: run `<power_cmd>` at the current `n` and the assumed effect.
- [ ] **Solve N**: if power `< <target_power>`, re-run at larger `n` (step up, then bisect) until it clears.
- [ ] **Audit validity**: list every flaw from the checklist below.
- [ ] **Revise**: fix the highest-priority flaw, set `n` to the power-adequate value, update `<design_doc>` (+ Preregistration section).
- [ ] Append a ledger row; stop when power clears the target **and** no flaws remain, or at `<budget>`.

**Iteration 0 — draft.** Write a first design to `<design_doc>`: the arms/conditions, the unit of
analysis and how units are assigned, the primary outcome and the exact planned test, the assumed effect
size (from `<outcome>`), and a first sample-size guess. Record nothing as final yet.

**Then, until stop (power met + no flaws, or budget):**

1. **Simulate power.** Run `<power_cmd>` at the current per-group `n` and the assumed effect, with the
   `--design` matching the planned test. Record the achieved power.
2. **Solve N.** If `power < <target_power>`, re-run the simulation at larger `n` — step up (e.g. double),
   then bisect — until power clears the target, and adopt that `n`.
3. **Audit validity.** Check the design against the checklist and list every flaw found:
   - **Confounding / no control** — is there a concurrent control group, or is the comparison against a
     historical/other-source baseline that differs in other ways?
   - **Randomization** — are units randomly assigned? If not, selection bias threatens any effect.
   - **Selection / sampling** — is the sample representative of the population the claim is about?
   - **Multiple comparisons** — more than one outcome/subgroup tested without correction?
   - **Optional stopping / peeking** — is there a pre-specified stopping rule, or will analysis run
     repeatedly until significant?
   - **Outcome & analysis pre-specification** — are the primary outcome and its single planned test
     fixed in advance (not chosen after seeing data)?
   - **Measurement** — is the outcome measured reliably and blind to condition where possible?
4. **Revise.** Fix the highest-priority flaw (or a tightly-coupled pair that cannot be fixed
   independently, such as adding a concurrent control and randomizing assignment to it) and set `n` to
   the power-adequate value. Update `<design_doc>`, including a **Preregistration** section: hypothesis,
   primary outcome, the one planned analysis, sample size + how it was derived, randomization scheme, and
   the stopping rule.
5. **Log** one ledger row and continue.

**Stop** when `power ≥ <target_power>` **and** the flaw list is empty, or at `<budget>`. Report the
final design + preregistration, the achieved power and required `n`, and — if stopping on budget — the
flaws still outstanding.

## Ledger
`<sandbox_root>/ledger.tsv`, tab-separated, never commas in the text. Header:
```
iter	n_per_group	power	open_flaws	change
```
Example:
```
iter	n_per_group	power	open_flaws	change
0	50	0.50	2	draft: volunteers vs last-year cohort, n=50
1	100	0.80	1	solved n for 80% power at d=0.4
2	100	0.80	0	randomized concurrent control; pre-specified single primary outcome + stopping rule
```
Report the **best** iteration: the final design, the achieved power and required `n`, and any flaws
still open if stopping on budget.

## Constraints
- **Power is computed at the minimal effect of interest**, not an optimistic one, because a design
  powered for an effect bigger than reality detects nothing real — and the `--design` in the simulation
  must match the test named in the design. Do not edit `tools/power_sim.py`.
- **A design does not pass on power alone** — an adequately powered but confounded or non-randomized
  design still fails; both gates (power and flaws) must clear.
- **Preregister before data**, so the eventual test is confirmatory rather than chosen after seeing
  results: the analysis, outcome, sample size, and stopping rule are fixed in advance.
- **One primary outcome and one planned test** drive the power and the verdict; secondary analyses are
  labeled exploratory.
- The sandbox is self-contained — no `../` escapes. Do not pause the loop to ask whether to continue.

© gaasher · MIT · gaasher/Agent-Loop-Skills

Similar skills

All skills for AI and ML →

power-analysis by other authors comes first.

power-analysis Claude Code Academic Workflow power-analysis works out statistical power, the sample size you need, and the minimum detectable effect for a study design, then drafts a registry-ready power section. CX Codex +1 ★ 1,729repo stars quantitative-analysis poemswe quantitative-analysis is a Claude Code skill for selecting statistical tests, calculating effect sizes, and analyzing power in numerical-data work. CX Codex +1 280installs ads-test AgriciDaniel ads-test is a skill for Claude Code, Codex, Cursor and other platforms that guides paid-media experiment design and readout, from hypothesis to decision. CX Codex +1 4,000installs ab-test-analysis VoltAgent ab-test-analysis evaluates A/B test data, inspects statistical and practical significance, and delivers principled ship or no-ship decisions. CX Codex ★ 6,505repo stars statistical-power K-Dense-AI statistical-power is a Claude Code skill that calculates the sample size needed to detect a meaningful effect. Supports protocol planning, grant submissions, and pre-registration. CX Codex +1 1,600installs research-data-analysis hanhuark research-data-analysis gives the agent a routine for baseline cases, hypothesis-driven DOE and plots that show uncertainty. Built for engineering experiments and simulations. CX Codex +1 ★ 26repo stars statistical-analyst Alireza Rezvani The statistical-analyst plugin for Claude Code runs hypothesis tests, evaluates A/B experiments, estimates sample sizes, and calculates confidence intervals. ★ 30,251repo stars

From the same set26

The whole set →
data-analysis The data-analysis skill explores a dataset by testing hypotheses and writes up findings backed by reproducible numbers. Use it for open-ended discovery, not to diagnose a known failure. CX Codex +1 ★ 185repo stars red-team red-team finds weaknesses in content filters, classifiers and APIs through adversarial stress-testing — generates attack inputs and builds a catalogue of failures. CX Codex +1 ★ 185repo stars prompt-optimize prompt-optimize is a Claude Code skill that iteratively refines an existing prompt against your evaluation command, keeping only edits that improve the score. CX Codex +1 ★ 185repo stars anomaly-investigation anomaly-investigation is a Claude Code skill that investigates a known spike, drop, or outlier by testing possible causes against data. CX Codex +1 ★ 185repo stars claim-verify claim-verify is a Claude Code skill that checks data-backed claims in draft reports against raw datasets. It recalculates metrics and stress-tests fragile conclusions. CX Codex +1 ★ 185repo stars literature-survey literature-survey is a Claude Code skill that maps research sources against claims. It runs for questions that need a cited survey of agreement, disputes, and gaps. CX Codex +1 ★ 185repo stars optimize-loop optimize-loop is a skill that runs an edit-measure-revert loop for code modules or SQL queries, keeping changes only when correctness holds and a cost metric falls. CX Codex +1 ★ 185repo stars research-proposal Refines an existing research proposal through iterative literature checks, rubric-based scoring, and targeted revisions until it passes. CX Codex +1 ★ 185repo stars

Project sponsors

Companies that keep this analytics open