Skip to main content
skill CX Codex

product-analytics · yonatangross — Claude Code skill: what it does, how to install

The product-analytics skill evaluates A/B tests, cohort retention, and funnels. It helps explain adoption changes, conversion losses, and experiment outcomes.

MIT 120 installs ★ 305 updated 2026-11-14

Install into Claude Code · Skills CLI (Vercel) All methods and platforms ↓
npx skills add yonatangross/orchestkit --skill product-analytics -a claude-code
Open on GitHub

What product-analytics does

The skill turns product metrics into a SHIP, EXTEND, or KILL decision. It compares A/B variants, works with lift, p-values, confidence intervals, MDE, and sample-size planning. It also covers day-N cohort retention and maps funnel stages to locate the largest loss.

Its guidance is split across rules/ab-test-evaluation.md, rules/cohort-retention.md, and rules/funnel-analysis.md, with formulas and test guidance in references/stats-cheat-sheet.md. The available tools are Read, Glob, Grep, WebFetch, and WebSearch. It also addresses peeking, multiple comparisons, sample ratio mismatch, novelty effects, and Simpson’s paradox. An SRM above 1% means the variant split is broken, so the assignment issue comes before result analysis.

What it helps with

  • compare A/B variants and choose a rollout decision
  • calculate retention for a selected cohort day
  • locate the largest conversion loss in a funnel
  • interpret p-values, MDE, and confidence intervals
  • detect sample ratio mismatch in an experiment

Example prompts for the agent

  • Review this A/B test for the new checkout flow. Compare control with treatment, calculate lift and p-value, then recommend SHIP, EXTEND, or KILL.
  • Write a SQL query for day-7 cohort retention using the user_activity table.
  • Map the signup funnel, identify its largest drop-off, and rank the stages for investigation.

How to install product-analytics

1Skills CLI (Vercel)

npx skills add yonatangross/orchestkit --skill product-analytics -a claude-code

2Manually: put the files in place

~/.claude/skills/product-analytics/SKILL.md

Copy the whole skill folder; for one project use .claude/skills/product-analytics/

Step by step, with commands for each platform: How to install skills in Claude Code: five ways

Questions about product-analytics

How do I install product-analytics in Claude Code?

Run npx skills add yonatangross/orchestkit --skill product-analytics -a claude-code. The skill file will be placed at ~/.claude/skills/product-analytics/SKILL.md.

What does product-analytics do?

It guides A/B-test evaluation, cohort retention analysis, funnel diagnosis, and statistical interpretation. The analysis ends with a SHIP, EXTEND, or KILL recommendation grounded in the measured results.

What should I do when an experiment has sample ratio mismatch?

Treat the experiment as INVALID, fix the assignment problem, and restart it. The rule uses a difference above 1% from the expected group split as the SRM threshold.

Source text

SKILL.md · MIT
---
name: product-analytics
license: MIT
compatibility: "Claude Code 2.1.277+."
description: "A/B test evaluation, cohort retention, funnels and significance. Use when analyzing experiments, feature adoption or conversion drop-offs."
user-invocable: false
disable-model-invocation: false
metadata:
  owner-agent: "product-strategist"
  category: document-asset-creation
  version: "1.0.0"
  author: "OrchestKit"
  complexity: "medium"
  tags: "ab-test, cohort, retention, funnel, conversion, analytics, experiment, statistical-significance"
allowed-tools: "Read Glob Grep WebFetch WebSearch"
---

# Product Analytics

Frameworks for turning raw product data into ship/extend/kill decisions. Covers A/B testing, cohort retention, funnel analysis, and the statistical foundations needed to make those decisions with confidence.

## Quick Reference

| Category | Rules | Impact | When to Use |
|----------|-------|--------|-------------|
| [A/B Test Evaluation](#ab-test-evaluation) | 1 | HIGH | Comparing variants, measuring significance, shipping decisions |
| [Cohort Retention](#cohort-retention) | 1 | HIGH | Feature adoption curves, day-N retention, engagement scoring |
| [Funnel Analysis](#funnel-analysis) | 1 | HIGH | Drop-off diagnosis, conversion optimization, stage mapping |
| [Statistical Foundations](#statistical-foundations) | 1 | HIGH | p-value interpretation, sample sizing, confidence intervals |

**Total: 4 rules across 4 categories**

## A/B Test Evaluation

Load `rules/ab-test-evaluation.md` for the full framework. Quick pattern:

```markdown
## Experiment: [Name]

Hypothesis: If we [change], then [primary metric] will [direction] by [amount]
  because [evidence or reasoning].

Sample size: [N per variant] — calculated for MDE=[X%], power=80%, alpha=0.05
Duration: [Minimum weeks] — never stop early (peeking bias)

Results:
  Control:   [metric value]  n=[count]
  Treatment: [metric value]  n=[count]
  Lift:      [+/- X%]        p=[value]  95% CI: [lower, upper]

Decision: SHIP / EXTEND / KILL
  Rationale: [One sentence grounded in numbers, not gut feel]
```

**Decision rules:**
- **SHIP** — p < 0.05, CI excludes zero, no guardrail regressions
- **EXTEND** — trending positive but underpowered (add runtime, not reanalysis)
- **KILL** — null result or guardrail degradation

See `rules/ab-test-evaluation.md` for sample size formulas, SRM checks, and pitfall list.

## Cohort Retention

Load `rules/cohort-retention.md` for full methodology. Quick pattern:

```sql
-- Day-N retention cohort query
SELECT
  DATE_TRUNC('week', first_seen)  AS cohort_week,
  COUNT(DISTINCT user_id)         AS cohort_size,
  COUNT(DISTINCT CASE
    WHEN activity_date = first_seen + INTERVAL '7 days'
    THEN user_id END) * 100.0
    / COUNT(DISTINCT user_id)     AS day_7_retention
FROM user_activity
GROUP BY 1
ORDER BY 1;
```

**Retention benchmarks (SaaS):**
- Day 1: 40–60% is healthy
- Day 7: 20–35% is healthy
- Day 30: 10–20% is healthy
- Flat curve after day 30 = product-market fit signal

See `rules/cohort-retention.md` for behavior-based cohorts, feature adoption curves, and engagement scoring.

## Funnel Analysis

Load `rules/funnel-analysis.md` for full methodology. Quick pattern:

```markdown
## Funnel: [Name] — [Date Range]

Stage 1: [Aware / Land]     → [N] users    (entry)
Stage 2: [Activate / Sign]  → [N] users    ([X]% from stage 1)
Stage 3: [Engage / Use]     → [N] users    ([X]% from stage 2)  ← biggest drop
Stage 4: [Convert / Pay]    → [N] users    ([X]% from stage 3)

Overall conversion: [X]%
Biggest drop-off:  Stage 2→3 ([X]% loss) — investigate first
```

**Optimization order:** Fix the largest drop-off first. A 5-point improvement at a high-volume step is worth more than a 20-point improvement at a low-volume step.

See `rules/funnel-analysis.md` for segmented funnels, micro-conversion tracking, and prioritization patterns.

## Statistical Foundations

Plain-English explanations of the stats every PM needs. Load `references/stats-cheat-sheet.md` for formulas and quick lookups.

**p-value in plain English:** The probability that you would see a result this extreme (or more extreme) if the change had zero effect. p=0.03 means a 3% chance you're looking at random noise. It does NOT mean "97% probability the change works."

**Confidence interval in plain English:** The range where the true effect probably lives. "Lift = +8%, 95% CI [+2%, +14%]" means you are fairly confident the real lift is somewhere between 2% and 14%. If the CI includes zero, you cannot claim a win.

**Minimum Detectable Effect (MDE):** The smallest lift you care about detecting. Setting MDE too small forces impractically large sample sizes. Anchor MDE to business value — if a 2% lift is not worth shipping, set MDE = 5%.

**Statistical vs practical significance:** A result can be statistically significant (p < 0.05) but practically meaningless (lift = 0.01%). Always check both. A 0.01% lift that costs 6 weeks of eng time is not a win.

## Common Pitfalls

1. **Peeking** — stopping an experiment early because results look good inflates false-positive rate. Commit to a runtime before launch.
2. **Multiple comparisons** — testing 10 metrics at p < 0.05 means ~1 false positive by chance. Apply Bonferroni correction or pre-register your primary metric.
3. **Sample Ratio Mismatch (SRM)** — if variant group sizes differ from expected split by > 1%, your experiment is broken. Fix before analyzing results.
4. **Novelty effect** — new features get inflated engagement in week 1. Run experiments long enough to see settled behavior (minimum 2 full business cycles).
5. **Simpson's paradox** — aggregate results can reverse when segmented. Always check results by key segments (device, plan tier, geography).

## Ship / Extend / Kill Framework

| Signal | Decision | Action |
|--------|----------|--------|
| p < 0.05, CI excludes zero, guardrails green | SHIP | Full rollout, update success metrics |
| Positive trend, underpowered (p = 0.10–0.15) | EXTEND | Add runtime, do not peek again |
| p > 0.15, flat or negative | KILL | Revert, document learnings, re-hypothesize |
| Guardrail regression, any p-value | KILL | Immediate revert regardless of primary metric |
| SRM detected | INVALID | Fix assignment bug, restart experiment |

## Related Skills

- `ork:product-frameworks` — OKRs, KPI trees, RICE prioritization, PRD templates
- `ork-extras:monitoring-observability` — Metric definition, alerting, and drift monitoring
- `ork:brainstorm` — Generate hypotheses and experiment ideas
- `ork:assess` — Evaluate product quality and risks

## References

- `rules/ab-test-evaluation.md` — Hypothesis, sample size, significance, decision matrix
- `rules/cohort-retention.md` — Cohort types, retention curves, SQL patterns
- `rules/funnel-analysis.md` — Stage mapping, drop-off identification, optimization
- `references/stats-cheat-sheet.md` — Formulas, test selection, power analysis

---

**Version:** 1.0.0 (March 2026)

© OrchestKit · MIT · yonatangross/orchestkit

Similar skills

All skills for Data and databases →

product-analytics by other authors comes first.

product-analytics Alireza Rezvani The product-analytics skill helps define product KPIs, shape metric dashboards, and read cohort, retention, and feature adoption data. CX Codex +1 684installs product-analytics daffy0208 Lays out product analytics: picking a North Star metric, mapping a signup funnel, comparing cohorts and marking user actions as events. CX Codex +1 244installs product-analytics travisjneuman product-analytics architects event tracking, funnels, cohorts, and privacy-compliant instrumentation for PostHog, Amplitude, Mixpanel, and Segment. CX Codex +1 139installs product-analytics Paweł Huryn The product-analytics Claude Code skill configures behavioral analytics without analytics cookies. It activates for work involving Clarity or PostHog, session recordings, or heatmaps. CX Codex +1 ★ 741repo stars instrument-product-analytics PostHog The instrument-product-analytics Claude Code skill adds PostHog events to product code. Run it after building a feature or reviewing a PR; it can also set up the SDK when the project lacks it. CX Codex +1 912installs product-analytics.dataflow tt-a1i This prompt guides Archify through a product analytics data-flow diagram, from web and mobile events to dashboards and a model, with a distinct path for consent-gated identity data. CX Codex GP ChatGPT ★ 98,599repo stars configuring-experiment-analytics PostHog The configuring-experiment-analytics Claude Code skill configures analytics for PostHog experiments. It sets exposure criteria and metrics, then helps interpret experiment results. CX Codex +1 425installs cro-methodology wondelai cro-methodology audits landing pages and funnels for conversion leaks using the CRE framework. Designs evidence-backed A/B testing hypotheses based on customer objections. CX Codex +1 6,600installs product-tool-stack RefoundAI product-tool-stack is a Claude Code skill for choosing and putting a product tool stack into practice. It joins discussions about analytics, experimentation, and team workflows. CX Codex +1 380installs product-analyst ncklrs product-analyst is a Claude Code skill that helps choose product metrics, analyze funnels, and plan experiments. It connects product events to team decisions. CX Codex +1 207installs

From the same set86

The whole set →
competitive-analysis competitive-analysis generates Porter's Five Forces models, SWOT grids, competitor dossiers, and positioning maps. Runs when researching market viability or crafting sales battlecards. CX Codex +1 125installs visualize-plan visualize-plan turns a proposed change into before-and-after architecture views, a risk map, and an execution sequence. Use it to review a plan or changes on the current branch. CX Codex +1 124installs market-sizing market-sizing is a Claude Code skill that calculates TAM, SAM and SOM using top-down and bottom-up methods, then cross-validates them. It runs when sizing a market or preparing pitch slides. CX Codex +1 124installs ai-ui-generation ai-ui-generation — a skill for Claude Code and Codex that enforces rules for generating, reviewing, and integrating AI-built UI components. Triggers on UI component creation and MCP visual rendering. CX Codex +1 123installs testing-perf The testing-perf skill for Claude Code runs load and performance tests with k6 and Locust and parallelizes pytest with xdist. It activates when writing load tests or speeding up test suite runs. CX Codex +1 120installs chain-patterns chain-patterns — a set of patterns for multi-stage pipelines: MCP checks, handoff files, checkpoint recovery, isolated worktree agents, and cron monitoring. CX Codex +1 118installs design-to-code design-to-code — a skill for Claude Code that turns screenshots, descriptions, and URLs into React components. Assembles UI elements while reusing existing Storybook pieces and design tokens. CX Codex +1 116installs testing-unit The testing-unit skill provides patterns for unit tests, HTTP mocks, fixtures, and test data in Vitest, Jest, and pytest. It applies while writing tests or setting up test doubles. CX Codex +1 116installs

Project sponsors

Companies that keep this analytics open