scanning-experiments-with-replay-vision — Claude Code skill
The scanning-experiments-with-replay-vision skill for Claude Code sets up a PostHog Replay Vision scanner targeted at an experiment. It watches exposed user recordings and analyzes differences across test variants with an LLM.
no license 148 installs ★ 101 updated 2026-10-24
npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a claude-codeWhat scanning-experiments-with-replay-vision does
This skill provisions an experiment type Replay Vision scanner tied to a specific PostHog A/B test. It fetches the underlying feature flag configuration via experiment-get, sets up a unified prompt, and balances session sampling across variants using balance_variants.
The LLM receives the visual change description but never gets informed which variant the recorded visitor actually saw. This prevents model bias and exposes broken feature flag rollouts directly from user actions. Cohort filtering runs entirely server-side; defining custom exposure filters inside query triggers an API error.
Draft experiments lacking a launch date cannot be provisioned, and ended runs require a backfill. The scanner is initially created disabled so the analysis prompt can be tested against real sessions before credits are spent.
What it helps with
- scan session recordings for a specific PostHog experiment
- balance session sampling evenly between test and control cohorts
- draft unbiased LLM prompts that hide variant assignment from the model
- estimate Replay Vision credit usage before turning the scanner on
Example prompts for the agent
- Create a Replay Vision scanner for experiment 123 to review checkout drop-offs
- Scan session recordings from our onboarding experiment using vision AI
- Set up an experiment scanner to inspect user interactions across all test variants
How to install scanning-experiments-with-replay-vision
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a claude-code2Manually: put the files in place
~/.claude/skills/scanning-experiments-with-replay-vision/SKILL.mdCopy the whole skill folder; for one project use .claude/skills/scanning-experiments-with-replay-vision/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a codex2Manually: put the files in place
.agents/skills/scanning-experiments-with-replay-vision/SKILL.mdFor all projects use ~/.agents/skills/
1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a cursor2Manually: put the files in place
.cursor/skills/scanning-experiments-with-replay-vision/SKILL.mdCursor also reads .agents/skills/ and .claude/skills/
1GitHub CLI
gh skill install PostHog/ai-plugin skills/scanning-experiments-with-replay-vision2Manually: put the files in place
.github/skills/scanning-experiments-with-replay-vision/SKILL.md1Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a gemini-cli2Platform CLI
gemini skills install https://github.com/PostHog/ai-plugin --path skills/scanning-experiments-with-replay-visionManually: put the files in place
.devin/skills/scanning-experiments-with-replay-vision/SKILL.mdThe legacy .windsurf/skills/ path still works
Skills CLI (Vercel)
npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a opencodeClaude.ai
Zip the folder and upload it in Settings → Capabilities → Skills
Step by step, with commands for each platform: How to install skills in Claude Code: five ways
Questions about scanning-experiments-with-replay-vision
How to install scanning-experiments-with-replay-vision in Claude Code?
Run npx skills add PostHog/ai-plugin --skill scanning-experiments-with-replay-vision -a claude-code in your terminal.
How does the experiment scanner avoid evaluation bias?
The LLM receives context about what changed on screen, but PostHog conceals the visitor's assigned variant so the summary reflects actual screen activity rather than expected outcomes.
When will the scanner refuse to provision?
It fails on draft experiments without a set start date, because the PostHog API requires an active or completed rollout with existing exposure data.