Skip to main content
skill CX Codex

statistics-verifier — Claude Code skill: what it does, how to install

statistics-verifier audits statistical claims, verifies experimental methodology, and spots analytical bias in research. Useful for peer reviews, A/B testing checks, and data audits.

MIT 222 installs ★ 114 updated 2026-10-25

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add travisjneuman/.claude --skill statistics-verifier -a claude-code
Open on GitHub

What statistics-verifier does

The skill applies a five-step protocol to scrutinize research claims. It reviews source credibility, study methodology, baseline plausibility, and replication evidence. The framework catches subtle distortions. It highlights red flags such as unreported sample sizes, missing confidence intervals, and correlation masquerading as causation.

Errors fall into experimental design, execution, interpretation, and reporting. Critical flaws like p-hacking or selection bias invalidate entire datasets. The skill maps data types to suitable tests, distinguishing Mann-Whitney from independent t-tests or chi-square from Fisher's exact. A/B tests aiming for a 5% relative lift from a 10% baseline conversion require roughly 3,200 users per group.

What it helps with

  • Audit statistical claims in preprints or whitepapers against raw data
  • Detect analytical errors like p-hacking and unadjusted multiple testing
  • Pick appropriate hypothesis tests based on data type and distribution
  • Verify whether an A/B test or survey sample size has sufficient statistical power

Example prompts for the agent

  • Check this experiment report for p-hacking and multiple testing issues.
  • Which test should I run to compare two independent groups with continuous data?
  • Validate our A/B test sample size: 10% baseline conversion with a target 5% relative lift.

How to install statistics-verifier

1Skills CLI (Vercel)

npx skills add travisjneuman/.claude --skill statistics-verifier -a claude-code

2Manually: put the files in place

~/.claude/skills/statistics-verifier/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/statistics-verifier/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about statistics-verifier

How to install statistics-verifier in Claude Code?

Run npx skills add travisjneuman/.claude --skill statistics-verifier -a claude-code. You can also manually add the instruction file to ~/.claude/skills/statistics-verifier/SKILL.md.

What does statistics-verifier do?

It provides structured checklists to audit data claims, assess study design, and spot analytical fallacies. It also guides test selection and calculates baseline sample sizes for experiments and surveys.

Which multiple comparisons corrections does this skill include?

The guide includes Bonferroni for conservative control, Holm-Bonferroni, Benjamini-Hochberg for false discovery rate management, Tukey's HSD for post-ANOVA pairwise tests, and Dunnett's method for control comparisons.

Source text

SKILL.md · MIT
---
name: statistics-verifier
description: Verify statistics from raw data with methodology checking, significance testing, claim validation, and bias detection. Use when fact-checking statistical claims, validating research findings, or auditing data analysis.
---

# Statistics Verifier

Structured frameworks for verifying statistical claims, validating research methodology, and detecting analytical errors and biases.

## Statistical Claim Verification Checklist

### Rapid Claim Assessment

```
CLAIM VERIFICATION PROTOCOL:

1. SOURCE CHECK
   - Who made the claim?
   - What is their expertise and incentive?
   - Where was it published (peer-reviewed, preprint, press release)?
   - Is the original data or study accessible?

2. METHODOLOGY CHECK
   - What type of study (RCT, observational, survey, meta-analysis)?
   - What was the sample size and population?
   - What was the measurement method?
   - Is the statistical test appropriate for the data type?

3. NUMBER SENSE CHECK
   - Does the claim pass a basic plausibility test?
   - Are units and denominators clearly stated?
   - Absolute vs relative numbers — which is being used?
   - Is the base rate provided for context?

4. REPLICATION CHECK
   - Have other studies found similar results?
   - Are the findings consistent across populations?
   - Has anyone attempted and failed to replicate?

5. CONCLUSION CHECK
   - Does the conclusion follow from the data?
   - Are alternative explanations addressed?
   - Is the scope of the claim proportional to the evidence?
```

### Claim Red Flags

| Red Flag | What It Means | Action |
| --- | --- | --- |
| No sample size given | Cannot assess reliability | Request or estimate N |
| Only relative risk reported | May hide small absolute effect | Calculate absolute difference |
| "Up to X%" framing | Cherry-picked best case | Ask for median or mean |
| No confidence interval | Precision unknown | Treat with skepticism |
| Correlation stated as causation | Confounders likely ignored | Check study design |
| Self-selected sample | Selection bias likely | Note limitation |
| Composite endpoint | May mask weak individual results | Decompose the endpoint |
| Subgroup analysis highlighted | Likely post-hoc fishing | Require pre-registration |

## Common Statistical Errors

### Error Detection Framework

```
CATEGORY 1: DESIGN ERRORS
- Sampling bias (convenience, voluntary response, survivorship)
- Confounding variables not controlled
- Insufficient sample size (underpowered study)
- No control group or inappropriate comparator
- Measurement instrument not validated

CATEGORY 2: ANALYSIS ERRORS
- Multiple comparisons without correction (p-hacking)
- Treating ordinal data as interval
- Assuming normality without checking
- Ignoring missing data patterns (MCAR vs MNAR)
- Using parametric tests on non-parametric data

CATEGORY 3: INTERPRETATION ERRORS
- Confusing statistical significance with practical significance
- Interpreting non-significant result as "no effect"
- Ecological fallacy (group-level applied to individuals)
- Simpson's paradox not checked
- Ignoring effect size and confidence intervals

CATEGORY 4: REPORTING ERRORS
- Selective reporting of favorable results
- Omitting negative or null findings
- Misleading axis scales in visualizations
- Presenting percentages without base numbers
- Switching between absolute and relative metrics
```

### Error Severity Assessment

| Error Type | Severity | Impact on Conclusion |
| --- | --- | --- |
| P-hacking / HARKing | Critical | Invalidates findings |
| Selection bias | Critical | Fundamentally flawed sample |
| Confounding not addressed | High | Alternative explanations remain |
| Wrong statistical test | High | Results may be artifactual |
| Multiple comparisons uncorrected | High | Inflated false positive rate |
| Small sample without power analysis | Medium | May miss real effects |
| Missing confidence intervals | Medium | Cannot judge precision |
| Misleading visualization | Medium | Misrepresents magnitude |
| Minor rounding errors | Low | Minimal impact |

## Significance Testing Framework

### Test Selection Guide

```
CHOOSING THE RIGHT TEST:

DATA TYPE → COMPARISON → TEST

Continuous + 2 groups + independent → Independent t-test (or Mann-Whitney)
Continuous + 2 groups + paired     → Paired t-test (or Wilcoxon signed-rank)
Continuous + 3+ groups + independent → One-way ANOVA (or Kruskal-Wallis)
Continuous + 2+ factors            → Two-way ANOVA (or Friedman)
Continuous + continuous             → Pearson correlation (or Spearman)

Categorical + 2 groups             → Chi-square test (or Fisher's exact)
Categorical + ordered              → Cochran-Armitage trend test
Binary outcome + predictors        → Logistic regression

Time-to-event + groups             → Log-rank test / Cox regression
Count data                          → Poisson regression
Proportion + large sample           → Z-test for proportions
```

### P-Value Interpretation Guide

```
P-VALUE CONTEXT:

p-value = P(data this extreme | null hypothesis is true)

COMMON MISINTERPRETATIONS:
  p = 0.03 does NOT mean:
  - "There is a 3% chance the result is due to chance"
  - "There is a 97% probability the hypothesis is true"
  - "The effect is large or important"
  - "The study will replicate"

  p = 0.03 DOES mean:
  - If the null hypothesis were true, data this extreme
    would occur about 3% of the time by chance alone.

THRESHOLDS (conventional, not absolute):
  p < 0.001  — strong evidence against null
  p < 0.01   — moderate evidence against null
  p < 0.05   — conventional threshold (context-dependent)
  p > 0.05   — insufficient evidence to reject null
                (NOT evidence of no effect)

ALWAYS COMPLEMENT WITH:
  - Effect size (Cohen's d, odds ratio, etc.)
  - Confidence interval (range of plausible values)
  - Practical significance (is the effect meaningful?)
  - Study power (could it have detected a real effect?)
```

### Multiple Comparisons Correction

| Method | When to Use | Conservativeness |
| --- | --- | --- |
| **Bonferroni** | Few comparisons, need strong control | Very conservative |
| **Holm-Bonferroni** | Moderate comparisons, step-down | Less conservative |
| **Benjamini-Hochberg** | Many comparisons (FDR control) | Liberal |
| **Tukey's HSD** | All pairwise comparisons after ANOVA | Moderate |
| **Dunnett's** | Multiple treatments vs one control | Moderate |

## Sample Size Validation

### Quick Reference Table

```
MINIMUM SAMPLE SIZE GUIDELINES:

Survey (population estimate):
  ±3% margin, 95% CI → n ≈ 1,067
  ±5% margin, 95% CI → n ≈ 385
  ±10% margin, 95% CI → n ≈ 97

A/B Test (detecting 5% relative lift):
  Baseline 10% conversion → n ≈ 3,200 per group
  Baseline 5% conversion  → n ≈ 6,400 per group
  Baseline 2% conversion  → n ≈ 16,000 per group

Clinical trial (medium effect d=0.5):
  Two-group comparison, 80% power → n ≈ 64 per group
  Two-group comparison, 90% power → n ≈ 86 per group

Correlation (detecting r=0.3):
  80% power, alpha=0.05 → n ≈ 85
  90% power, alpha=0.05 → n ≈ 113
```

### Power Analysis Checklist

| Parameter | Must Specify | Source |
| --- | --- | --- |
| Alpha (Type I error rate) | Yes | Convention (usually 0.05) |
| Power (1 - Type II error) | Yes | Usually 0.80 or 0.90 |
| Effect size | Yes | Prior research or MCID |
| Variance / SD | Yes | Pilot data or literature |
| Sample size | Calculated | Output of power analysis |
| Attrition rate | Recommended | Inflate N by expected dropout |

## Correlation vs Causation Checklist

### Bradford Hill Criteria for Causation

```
DOES CORRELATION IMPLY CAUSATION? CHECK:

1. STRENGTH           Is the association large?
                      Larger effects harder to explain away.

2. CONSISTENCY        Replicated across settings, populations?
                      Multiple studies, same finding.

3. SPECIFICITY        Is X linked specifically to Y (not everything)?
                      Less useful for multifactorial diseases.

4. TEMPORALITY        Does X precede Y in time?
                      REQUIRED — cause must come before effect.

5. BIOLOGICAL GRADIENT  Does more X produce more Y (dose-response)?
                        Strong support for causation.

6. PLAUSIBILITY       Is there a credible mechanism?
                      Based on current knowledge.

7. COHERENCE          Consistent with known biology/theory?
                      No conflict with established facts.

8. EXPERIMENT         Does removing X reduce Y?
                      Strongest evidence (RCT).

9. ANALOGY            Similar exposures cause similar effects?
                      Weakest criterion, supporting only.

VERDICT:
  Criteria 1-3 met + Temporality → Suggestive of causation
  Criteria 1-6 met + Experiment  → Strong evidence of causation
  Only correlation observed      → Association only, cannot infer cause
```

### Common Third-Variable Confounders

| Observed Association | Likely Confounder |
| --- | --- |
| Ice cream sales and drowning | Warm weather (season) |
| Shoe size and reading ability | Age |
| Hospital visits and death rate | Illness severity |
| Organic food and health | Socioeconomic status |
| Screen time and depression | Social isolation, sleep |

## Survey Methodology Review

### Survey Quality Assessment

```
SURVEY METHODOLOGY CHECKLIST:

SAMPLING:
- [ ] Probability sampling method described?
- [ ] Sampling frame defined and appropriate?
- [ ] Response rate reported (acceptable: >60% mail, >80% in-person)?
- [ ] Non-response bias assessed?

QUESTIONNAIRE:
- [ ] Questions validated or adapted from validated instruments?
- [ ] Leading or double-barreled questions absent?
- [ ] Response options balanced and exhaustive?
- [ ] Pilot tested with target population?

ADMINISTRATION:
- [ ] Mode (online, phone, in-person) appropriate?
- [ ] Anonymity/confidentiality assured?
- [ ] Informed consent obtained?
- [ ] Social desirability bias mitigated?

ANALYSIS:
- [ ] Weighting applied for non-response or oversampling?
- [ ] Margin of error and confidence level reported?
- [ ] Subgroup analyses pre-specified (not exploratory)?
```

## Data Visualization Integrity Checks

### Chart Audit Checklist

| Check | What to Look For | Fail Condition |
| --- | --- | --- |
| Y-axis starts at zero (bar charts) | Truncated axis exaggerates differences | Axis starts above zero without clear label |
| Consistent scale | Both axes have proportional increments | Non-linear scale without explanation |
| Area proportional to data | Bubble/icon size matches values | Area misrepresents magnitude |
| Time axis evenly spaced | Equal intervals between data points | Uneven spacing compresses/expands trends |
| Appropriate chart type | Data type matches visualization | Pie chart with 20+ categories |
| Context provided | Benchmarks, comparisons, baselines | Single data point with no reference |
| Source cited | Data origin traceable | No source attribution |
| Dual axes used responsibly | Two Y-axes can create false correlations | Arbitrary scaling implies relationship |

### Misleading Visualization Patterns

```
WATCH FOR THESE TRICKS:

1. TRUNCATED AXIS
   Small differences look dramatic when baseline removed.
   FIX: Always check if y-axis starts at zero for bar charts.

2. CHERRY-PICKED TIME WINDOW
   Start/end dates chosen to show desired trend.
   FIX: Ask for longer time series with consistent intervals.

3. 3D EFFECTS
   Perspective distortion makes sizes unequal.
   FIX: Use flat 2D charts for accurate comparison.

4. DUAL AXIS MANIPULATION
   Two y-axes scaled to create apparent correlation.
   FIX: Normalize data or use separate panels.

5. CUMULATIVE VS DAILY
   Cumulative charts always go up — hides declining rates.
   FIX: Show rate of change alongside cumulative.
```

## Bias Detection Framework

### Cognitive Biases in Data Analysis

```
BIAS DETECTION CHECKLIST:

CONFIRMATION BIAS
- Are they only presenting data that supports their hypothesis?
- Were negative results reported?
- Was the analysis plan pre-registered?

ANCHORING BIAS
- Is the first number presented influencing interpretation of later data?
- Are comparisons made to appropriate benchmarks?

SURVIVORSHIP BIAS
- Are only successful cases included (ignoring failures)?
- Is the denominator complete (not just survivors)?

AVAILABILITY BIAS
- Are dramatic or recent events overweighted?
- Is systematic data used rather than anecdotal evidence?

PUBLICATION BIAS
- Is there a funnel plot asymmetry (meta-analyses)?
- Are null results published or only significant ones?

TEXAS SHARPSHOOTER FALLACY
- Were clusters or patterns found after looking at data?
- Was the hypothesis formed before or after seeing results?
```

### Bias Severity Matrix

| Bias | Detection Method | Mitigation |
| --- | --- | --- |
| Selection bias | Compare sample to population demographics | Probability sampling, weighting |
| Measurement bias | Check instrument validity and calibration | Validated instruments, blinding |
| Reporting bias | Look for asymmetric funnel plots | Pre-registration, open data |
| Recall bias | Compare to objective records | Prospective data collection |
| Observer bias | Check if assessors were blinded | Double-blind design |
| Attrition bias | Compare completers vs dropouts | Intention-to-treat analysis |

## Reproducibility Checklist

### Study Reproducibility Assessment

```
REPRODUCIBILITY REQUIREMENTS:

DATA AVAILABILITY:
- [ ] Raw data accessible (repository, supplement, on request)?
- [ ] Data dictionary / codebook provided?
- [ ] Data collection protocol documented?

CODE / ANALYSIS:
- [ ] Analysis code shared (GitHub, OSF, supplement)?
- [ ] Software versions and packages specified?
- [ ] Random seeds set for reproducible computation?
- [ ] Pipeline documented end-to-end?

METHODOLOGY:
- [ ] Study pre-registered (OSF, ClinicalTrials.gov)?
- [ ] Deviations from protocol documented?
- [ ] All outcome measures reported (not just significant ones)?
- [ ] Sensitivity analyses included?

REPORTING:
- [ ] Follows reporting guidelines (CONSORT, STROBE, PRISMA)?
- [ ] Effect sizes and confidence intervals reported?
- [ ] Power analysis or sample size justification provided?
- [ ] Limitations section thorough and honest?
```

### Reporting Standards by Study Type

| Study Type | Guideline | Key Elements |
| --- | --- | --- |
| Randomized trial | CONSORT | Flow diagram, ITT analysis, blinding |
| Observational study | STROBE | Selection criteria, confounders, missing data |
| Systematic review | PRISMA | Search strategy, inclusion criteria, risk of bias |
| Diagnostic accuracy | STARD | Index test, reference standard, flow diagram |
| Qualitative research | COREQ | Research team, study design, data analysis |
| Prediction model | TRIPOD | Model development, validation, performance |

## Quick Verification Workflow

```
FAST VERIFICATION (5 minutes):

1. Read the claim carefully — what exactly is being stated?
2. Check: source, sample size, study type
3. Ask: absolute or relative? What is the base rate?
4. Check: confidence interval or margin of error given?
5. Search: has this been replicated independently?

VERDICT CATEGORIES:
  VERIFIED    — multiple strong sources, robust methodology
  PLAUSIBLE   — reasonable evidence, some limitations
  UNCERTAIN   — mixed evidence, methodology concerns
  MISLEADING  — technically true but presented deceptively
  FALSE       — contradicted by strong evidence
  UNVERIFIABLE — cannot assess with available information
```

## See Also

- [Data Science](../data-science/SKILL.md)
- [Research Presenter](../research-presenter/SKILL.md)
- [Product Analytics](../product-analytics/SKILL.md)

© travisjneuman · MIT · travisjneuman/.claude

Similar skills

All skills for Data and databases →

From the same set117

The whole set →
video-production video-production — a skill that walks a video from script and shot list through lighting, sound and codecs, then installs in Claude with a single command. CX Codex +1 237installs electron-desktop electron-desktop — a skill for building cross-platform desktop applications with Electron for Windows, macOS and Linux. CX Codex +1 236installs sustainability-esg sustainability-esg hands the agent ready ESG frameworks: a materiality matrix, carbon and energy goals, supplier assessment tiers, and GRI, SASB and TCFD reporting formats. CX Codex +1 225installs investment-memo-generator investment-memo-generator generates structured investment memos, due diligence checklists, and financial frameworks tailored to VC, growth, and private equity deals. CX Codex +1 223installs hr-talent hr-talent is a Claude Code skill that provides people operations frameworks for recruiting, employee performance reviews, and compensation planning. It structures talent management in prompt sessions. CX Codex +1 219installs embedded-iot embedded-iot — a skill for firmware work: GPIO, I2C and MQTT patterns, FreeRTOS tasks and power rules for ESP32, STM32 and Arduino boards. CX Codex +1 212installs contract-redliner contract-redliner is a Claude Code skill that reviews contract clauses, flags risks, and drafts edits. It activates for contract checks and negotiation prep. CX Codex +1 206installs devops-cloud devops-cloud — a skill for Claude Code and Codex that assists with cloud infrastructure and CI/CD pipelines. Triggers on tasks involving AWS, GCP, Azure, Kubernetes, or Terraform. CX Codex +1 204installs

Project sponsors

Companies that keep this analytics open